{"id":55,"date":"2024-05-12T18:39:51","date_gmt":"2024-05-12T18:39:51","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/?page_id=55"},"modified":"2024-12-05T21:57:54","modified_gmt":"2024-12-05T21:57:54","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/","title":{"rendered":"Method"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1042\" height=\"441\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-content\/uploads\/sites\/112\/2024\/12\/arch_new.png\" alt=\"\" class=\"wp-image-243\" style=\"width:675px;height:auto\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-content\/uploads\/sites\/112\/2024\/12\/arch_new.png 1042w, https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-content\/uploads\/sites\/112\/2024\/12\/arch_new-300x127.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-content\/uploads\/sites\/112\/2024\/12\/arch_new-1024x433.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-content\/uploads\/sites\/112\/2024\/12\/arch_new-768x325.png 768w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/figure>\n\n\n\n<p>Our method works with as few as 4 input images without camera pose information to directly predict 3D Gaussians&#8217; attributes that can then be rendered from any novel view using rasterization. It also predicts camera rays supervised on ground-truth camera pose information. The following steps outline our approach:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Data preprocessing:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Rescale each scene so that each view is within a certain distance (unit distance) from the object of interest. We use triangulation to find the point of intersection of all cameras&#8217; optical axes and use this to perform the rescaling.<\/li>\n\n\n\n<li>Randomly select one of the camera frames as the world frame and redefine all information (camera information if available) w.r.t. to this selected world frame so that this selected camera&#8217;s center is now the world origin and the camera has identity rotation. <\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Feature Extraction:<\/strong>\n<ul class=\"wp-block-list\">\n<li>We use the pre-trained <a href=\"https:\/\/dinov2.metademolab.com\/\">DINOv2<\/a> model to extract patch-level features for all input images.<\/li>\n\n\n\n<li>We concatenate sinusoidal image positional embeddings to differentiate the first view (which is considered the world frame) from the rest.<\/li>\n\n\n\n<li>We concatenate patch-level sinusoidal positional embeddings similar to ViT to differentiate between different patches within each input image.<\/li>\n\n\n\n<li>If camera information is available in the form of a projection matrix (a combination of intrinsic and extrinsic information), we concatenate sinusoidal positional embeddings of these matrices for each image.<\/li>\n\n\n\n<li>We pass the final patch-level features of all images through a self-attention layer to implicitly find correspondences across images and share multi-view information.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Transformer Blocks:<\/strong>\n<ul class=\"wp-block-list\">\n<li>We randomly initialize a set of latent vectors where each vector is used to predict K Gaussians towards the end of the pipeline.<\/li>\n\n\n\n<li>These latent vectors and the previously extracted input images&#8217; features are passed through several transformer blocks consisting of self-attention and cross-attention layers to adequately condition the latent Gaussian vectors for the final predictions.<\/li>\n\n\n\n<li>Similar to the above transformer blocks there are a few self-attention transformer blocks that take in image features and predict the camera pose as rays.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Rendering:<\/strong>\n<ul class=\"wp-block-list\">\n<li>In the end, we pass the latent Gaussian vectors through a linear layer followed by appropriate activation functions to predict the respective Gaussian attributes of opacity, scale, rotation, means, and RGB. These Gaussians are defined w.r.t. the first view&#8217;s frame.<\/li>\n\n\n\n<li>We use the rasterizer from vanilla 3DGS to render these predicted Gaussians from a given view.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Loss:<\/strong>\n<ul class=\"wp-block-list\">\n<li>We apply an initialization loss directly on the predicted Gaussians&#8217; means and scales for about 1.5k iterations to make sure the network has a good initialization.<\/li>\n\n\n\n<li>After this phase, we apply an RGB loss (MSE) on the renderings obtained from the predicted Gaussians. We use the 4 input views and 2 additional unseen views for this loss.<\/li>\n\n\n\n<li>During the above phase, a parallel branch takes in the image features and predicts camera rays and an MSE loss is applied between the predicted rays and GT rays.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Inference:<\/strong> Once trained on a large-scale dataset, our pipeline just needs to be fed a few input images (as few as 4) to predict a 3D representation in the form of 3D Gaussians in a single feed-forward pass without any scene-specific optimization, unlike vanilla 3DGS. We also predict camera rays in parallel which are then used to further refine the output fidelity using test-time adaptation.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Our method works with as few as 4 input images without camera pose information to directly predict 3D Gaussians&#8217; attributes that can then be rendered from any novel view using rasterization. It also predicts camera rays supervised on ground-truth camera pose information. The following steps outline our approach:<\/p>\n","protected":false},"author":218,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-55","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Method - Sparse view 3D Pose and Reconstruction<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - Sparse view 3D Pose and Reconstruction\" \/>\n<meta property=\"og:description\" content=\"Our method works with as few as 4 input images without camera pose information to directly predict 3D Gaussians&#8217; attributes that can then be rendered from any novel view using rasterization. It also predicts camera rays supervised on ground-truth camera pose information. The following steps outline our approach:\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/\" \/>\n<meta property=\"og:site_name\" content=\"Sparse view 3D Pose and Reconstruction\" \/>\n<meta property=\"article:modified_time\" content=\"2024-12-05T21:57:54+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-content\/uploads\/sites\/112\/2024\/12\/arch_new.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/method\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/method\\\/\",\"name\":\"Method - Sparse view 3D Pose and Reconstruction\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/method\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/method\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/wp-content\\\/uploads\\\/sites\\\/112\\\/2024\\\/12\\\/arch_new.png\",\"datePublished\":\"2024-05-12T18:39:51+00:00\",\"dateModified\":\"2024-12-05T21:57:54+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/method\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/method\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/method\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/wp-content\\\/uploads\\\/sites\\\/112\\\/2024\\\/12\\\/arch_new.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/wp-content\\\/uploads\\\/sites\\\/112\\\/2024\\\/12\\\/arch_new.png\",\"width\":1042,\"height\":441},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/method\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/\",\"name\":\"Sparse view 3D Pose and Reconstruction\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team14\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - Sparse view 3D Pose and Reconstruction","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/","og_locale":"en_US","og_type":"article","og_title":"Method - Sparse view 3D Pose and Reconstruction","og_description":"Our method works with as few as 4 input images without camera pose information to directly predict 3D Gaussians&#8217; attributes that can then be rendered from any novel view using rasterization. It also predicts camera rays supervised on ground-truth camera pose information. The following steps outline our approach:","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/","og_site_name":"Sparse view 3D Pose and Reconstruction","article_modified_time":"2024-12-05T21:57:54+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-content\/uploads\/sites\/112\/2024\/12\/arch_new.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/","name":"Method - Sparse view 3D Pose and Reconstruction","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-content\/uploads\/sites\/112\/2024\/12\/arch_new.png","datePublished":"2024-05-12T18:39:51+00:00","dateModified":"2024-12-05T21:57:54+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-content\/uploads\/sites\/112\/2024\/12\/arch_new.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-content\/uploads\/sites\/112\/2024\/12\/arch_new.png","width":1042,"height":441},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/method\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/","name":"Sparse view 3D Pose and Reconstruction","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-json\/wp\/v2\/pages\/55","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-json\/wp\/v2\/users\/218"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-json\/wp\/v2\/comments?post=55"}],"version-history":[{"count":9,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-json\/wp\/v2\/pages\/55\/revisions"}],"predecessor-version":[{"id":258,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-json\/wp\/v2\/pages\/55\/revisions\/258"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team14\/wp-json\/wp\/v2\/media?parent=55"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}