{"id":18,"date":"2024-04-29T22:50:02","date_gmt":"2024-04-29T22:50:02","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/?page_id=18"},"modified":"2024-12-15T05:45:49","modified_gmt":"2024-12-15T05:45:49","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/","title":{"rendered":"Method"},"content":{"rendered":"
\n
\"\"<\/a><\/figure>\n<\/div>\n\n\n

Our method is based on COLMAP. We use SuperPoint and SuperGlue for feature extraction and matching. We apply Pixel-Perfect refinement to COLMAP to refine the detected keypoints before triangulation and reconstructed 3D points in bundle adjustment. We also implement an intrinsics refinement stage after the bundle adjustment stage to improve the recovered camera intrinsics and reconstructed scene using ground-truth extrinsics.<\/p>\n\n\n\n

Input RGB Images<\/h2>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n

Input images are from Multiface dataset, which are captured by multiple cameras from multiple views.<\/p>\n\n\n\n

Feature Extraction<\/h2>\n\n\n\n

Firstly, 2D keypoints and their feature descriptors are extracted from the input images. We experimented with both traditional (e.g. SIFT) and learning-based (e.g. SuperPoint) feature extractors. Compared to SIFT, SuperPoint can yield more robust and accurate results.<\/p>\n\n\n\n

Feature Matching<\/h2>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n

The keypoints detected in the previous step are matched using feature matching algorithms. We experimented with both traditional (e.g. Nearest Neighbour) and learning-based (e.g. SuperGlue) matching algorithms. SuperGlue uses a Graph Neural Network with self- and cross-attention to improve feature representations, and yield better matching results.<\/p>\n\n\n\n

Keypoint Refinement<\/h2>\n\n\n\n

A dense feature map is computed for each input image by a deep CNN. Then, these feature maps are used to refined the 2D keypoints. Specifically, we use Pixel-Perfect to adjust the locations of 2D keypoints belonging to the same track by optimizing its featuremetric consistency along tentative matches with a cost function:<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Triangulation<\/h2>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n

In the triangulation stage, 3D points are computed from predicted camera intrinsics and extrinsics and keypoint positions.<\/p>\n\n\n\n

Bundle Adjustment<\/h2>\n\n\n\n

In the bundle adjustment stage, camera intrinsics and extrinsics and triangulated points are jointly optimized. We also use aforementioned dense feature maps produced by Pixel-Perfect. Specifically, they are refined using a featuremetric cost:<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Intrinsics Refinement<\/h2>\n\n\n\n

An additional refinement stage is applied to optimize intrinsic parameters using known ground-truth extrinsics.<\/p>\n\n\n\n

Instead of directly replacing estimated extrinsics with ground-truth values, we introduce an extrinsics regularization term with a gradually increasing coefficient during optimization. This approach converts the constraint from a hard replacement to a more flexible adjustment, thereby mitigating the risk of local minima.<\/p>\n\n\n\n

We also introduce an intrinsics variance term to jointly optimize intrinsics across all frames, resulting in a more global and consistent calibration. This contrasts with COLMAP and PixelSfM, where intrinsics are refined independently for each frame.<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

Our method is based on COLMAP. We use SuperPoint and SuperGlue for feature extraction and matching. We apply Pixel-Perfect refinement to COLMAP to refine the detected keypoints before triangulation and reconstructed 3D points in bundle adjustment. We also implement an intrinsics refinement stage after the bundle adjustment stage to improve the recovered camera intrinsics and … <\/p>\n

Continue reading “Method”<\/span><\/a><\/p>\n","protected":false},"author":200,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-18","page","type-page","status-publish","hentry"],"yoast_head":"\nMethod - Large Scale Camera Array Calibration via SfM<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - Large Scale Camera Array Calibration via SfM\" \/>\n<meta property=\"og:description\" content=\"Our method is based on COLMAP. We use SuperPoint and SuperGlue for feature extraction and matching. We apply Pixel-Perfect refinement to COLMAP to refine the detected keypoints before triangulation and reconstructed 3D points in bundle adjustment. We also implement an intrinsics refinement stage after the bundle adjustment stage to improve the recovered camera intrinsics and … Continue reading "Method"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/\" \/>\n<meta property=\"og:site_name\" content=\"Large Scale Camera Array Calibration via SfM\" \/>\n<meta property=\"article:modified_time\" content=\"2024-12-15T05:45:49+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-content\/uploads\/sites\/102\/2024\/12\/Pipeline-New-1024x297.jpg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/method\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/method\\\/\",\"name\":\"Method - Large Scale Camera Array Calibration via SfM\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/method\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/method\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/wp-content\\\/uploads\\\/sites\\\/102\\\/2024\\\/12\\\/Pipeline-New-1024x297.jpg\",\"datePublished\":\"2024-04-29T22:50:02+00:00\",\"dateModified\":\"2024-12-15T05:45:49+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/method\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/method\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/method\\\/#primaryimage\",\"url\":\"\",\"contentUrl\":\"\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/method\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/\",\"name\":\"Large Scale Camera Array Calibration via SfM\",\"description\":\"Students: Long V\u00e2n Tran Ha, Hewei Wang, Jinjiang You | Advisors: Ioannis Gkioulekas (CMU), Shubham Garg (Meta Reality Labs), Wei Pu (Meta Reality Labs)\",\"publisher\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/#organization\",\"name\":\"Large Scale Camera Array Calibration via SfM\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/wp-content\\\/uploads\\\/sites\\\/102\\\/2024\\\/12\\\/Meta-CMU-Logo.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/wp-content\\\/uploads\\\/sites\\\/102\\\/2024\\\/12\\\/Meta-CMU-Logo.png\",\"width\":1053,\"height\":250,\"caption\":\"Large Scale Camera Array Calibration via SfM\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team4\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - Large Scale Camera Array Calibration via SfM","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/","og_locale":"en_US","og_type":"article","og_title":"Method - Large Scale Camera Array Calibration via SfM","og_description":"Our method is based on COLMAP. We use SuperPoint and SuperGlue for feature extraction and matching. We apply Pixel-Perfect refinement to COLMAP to refine the detected keypoints before triangulation and reconstructed 3D points in bundle adjustment. We also implement an intrinsics refinement stage after the bundle adjustment stage to improve the recovered camera intrinsics and … Continue reading \"Method\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/","og_site_name":"Large Scale Camera Array Calibration via SfM","article_modified_time":"2024-12-15T05:45:49+00:00","og_image":[{"url":"http:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-content\/uploads\/sites\/102\/2024\/12\/Pipeline-New-1024x297.jpg","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/","name":"Method - Large Scale Camera Array Calibration via SfM","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/#primaryimage"},"thumbnailUrl":"http:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-content\/uploads\/sites\/102\/2024\/12\/Pipeline-New-1024x297.jpg","datePublished":"2024-04-29T22:50:02+00:00","dateModified":"2024-12-15T05:45:49+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/#primaryimage","url":"","contentUrl":""},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/method\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/","name":"Large Scale Camera Array Calibration via SfM","description":"Students: Long V\u00e2n Tran Ha, Hewei Wang, Jinjiang You | Advisors: Ioannis Gkioulekas (CMU), Shubham Garg (Meta Reality Labs), Wei Pu (Meta Reality Labs)","publisher":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/#organization","name":"Large Scale Camera Array Calibration via SfM","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/#\/schema\/logo\/image\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-content\/uploads\/sites\/102\/2024\/12\/Meta-CMU-Logo.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-content\/uploads\/sites\/102\/2024\/12\/Meta-CMU-Logo.png","width":1053,"height":250,"caption":"Large Scale Camera Array Calibration via SfM"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-json\/wp\/v2\/pages\/18","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-json\/wp\/v2\/users\/200"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-json\/wp\/v2\/comments?post=18"}],"version-history":[{"count":15,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-json\/wp\/v2\/pages\/18\/revisions"}],"predecessor-version":[{"id":188,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-json\/wp\/v2\/pages\/18\/revisions\/188"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team4\/wp-json\/wp\/v2\/media?parent=18"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}