{"id":17,"date":"2022-12-15T15:11:54","date_gmt":"2022-12-15T15:11:54","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/?page_id=17"},"modified":"2023-05-09T16:19:46","modified_gmt":"2023-05-09T16:19:46","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/","title":{"rendered":"Method"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Baseline<\/h2>\n\n\n\n<p>We use the approach mentioned in DynamicNeRF [1] as our baseline. The overall architecture of DynamicNeRF consists of two NeRF models, one for the static scene and the other for the dynamic object. Both of the NeRF models take the following inputs:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>images from different viewpoints and timesteps of the scene containing dynamic objects <\/li>\n\n\n\n<li>ground truth mask of dynamic objects in the scene<\/li>\n<\/ol>\n\n\n\n<p>The static model implements vanilla NeRF, querying color and density given the position and viewing direction. The dynamic model takes not only the position and viewpoint, but also the timestep to output the color and density as well as the blending factor at each point and timestep. The blending factor ranges between 0 to 1, denoting the importance of the prediction of the dynamic model at each point. The image is finally rendered using volumetric rendering.<\/p>\n\n\n\n<p>An overview of the DynamicNeRFs rendering pipeline is shown in the figure below.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"404\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/Screenshot-from-2022-12-20-23-28-15-1024x404.png\" alt=\"\" class=\"wp-image-82\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/Screenshot-from-2022-12-20-23-28-15-1024x404.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/Screenshot-from-2022-12-20-23-28-15-300x118.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/Screenshot-from-2022-12-20-23-28-15-768x303.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/Screenshot-from-2022-12-20-23-28-15-1536x606.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/Screenshot-from-2022-12-20-23-28-15.png 1920w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><figcaption class=\"wp-element-caption\">Approach used in DynamicNeRF [1]<\/figcaption><\/figure>\n\n\n\n<p>Some baseline results are shown in the videos below.<\/p>\n\n\n\n<figure class=\"wp-block-video aligncenter\"><video height=\"270\" style=\"aspect-ratio: 480 \/ 270;\" width=\"480\" controls src=\"http:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/balloon_scene.mp4\"><\/video><figcaption class=\"wp-element-caption\">DynamicNeRF on Balloon scene<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"270\" style=\"aspect-ratio: 480 \/ 270;\" width=\"480\" controls muted src=\"http:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/truck_scene.mp4\"><\/video><figcaption class=\"wp-element-caption\">DynamicNeRF on Truck scene<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Our Contribution<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Flexible multi-camera composition framework to add an arbitrary number of dynamic objects into a static 3D scene<\/li>\n\n\n\n<li>Novel-view augmentation (NOVA) strategy for learning better per-point blending factors<\/li>\n\n\n\n<li>Corresponding novel-view losses for high rendered image fidelity<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Multi-camera Composition<\/h3>\n\n\n\n<p>Our framework uses separate cameras to learn different parts of the scene. To specify what these cameras are responsible for, a segmentation mask is provided for each camera. Given the images at different time steps and these segmentation masks, each camera is assigned a corresponding NeRF that is static or dynamic based on the dynamicity of the scene parts it models (see figure below). <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"416\" src=\"http:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2023\/05\/nova_architecture-1024x416.jpg\" alt=\"\" class=\"wp-image-138\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2023\/05\/nova_architecture-1024x416.jpg 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2023\/05\/nova_architecture-300x122.jpg 300w, https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2023\/05\/nova_architecture-768x312.jpg 768w, https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2023\/05\/nova_architecture.jpg 1509w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/figure>\n\n\n\n<p>The static NeRF is time-invariant and can use images across all time steps to learn the scene, predicting the per-point color and density given the point&#8217;s position and viewing direction. The dynamic NeRF is time-varying and uses a single image per time-step to learn the scene at a particular timestep, predicting the per-point color, density, scene flow, and blending factor given the point&#8217;s position, viewing direction and time.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Novel-view augmentation (NOVA)<\/h3>\n\n\n\n<p>During training, we shift the camera responsible for the dynamic object to a novel view. Given the camera&#8217;s relative transformation, we calculate the ground truth segmentation mask at the novel view using stereo geometry. Points are sampled along the rays of the camera at the novel viewpoint C<sub>2<\/sub> and passed through the corresponding NeRF.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2023\/05\/novel_augmentation.png\" alt=\"\" class=\"wp-image-141\" width=\"493\" height=\"357\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2023\/05\/novel_augmentation.png 766w, https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2023\/05\/novel_augmentation-300x217.png 300w\" sizes=\"auto, (max-width: 493px) 100vw, 493px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Novel-view losses<\/h3>\n\n\n\n<p>We introduce a few losses to ensure high image fidelity when placing objects at novel points in the scene.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Novel-View Mask loss: MSE between GT and predicted masks<\/li>\n\n\n\n<li>Full Novel-View RGB loss: MSE between GT and predicted RGB image<\/li>\n\n\n\n<li>Blending loss: sum of blending factors across cameras should be equal to one<\/li>\n\n\n\n<li>Alpha loss: predicted alpha outside mask should be zero<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">References<\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Gao, Chen, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. &#8220;Dynamic view synthesis from dynamic monocular video.&#8221; In Proceedings of the IEEE\/CVF International Conference on Computer Vision, pp. 5712-5721. 2021.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>Baseline We use the approach mentioned in DynamicNeRF [1] as our baseline. The overall architecture of DynamicNeRF consists of two NeRF models, one for the static scene and the other for the dynamic object. Both of the NeRF models take the following inputs: The static model implements vanilla NeRF, querying color and density given the &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Method&#8221;<\/span><\/a><\/p>\n","protected":false},"author":145,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-17","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Method - Sensor Simulation with Dynamic Objects<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - Sensor Simulation with Dynamic Objects\" \/>\n<meta property=\"og:description\" content=\"Baseline We use the approach mentioned in DynamicNeRF [1] as our baseline. The overall architecture of DynamicNeRF consists of two NeRF models, one for the static scene and the other for the dynamic object. Both of the NeRF models take the following inputs: The static model implements vanilla NeRF, querying color and density given the &hellip; Continue reading &quot;Method&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/\" \/>\n<meta property=\"og:site_name\" content=\"Sensor Simulation with Dynamic Objects\" \/>\n<meta property=\"article:modified_time\" content=\"2023-05-09T16:19:46+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/Screenshot-from-2022-12-20-23-28-15-1024x404.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/method\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/method\\\/\",\"name\":\"Method - Sensor Simulation with Dynamic Objects\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/method\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/method\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/wp-content\\\/uploads\\\/sites\\\/72\\\/2022\\\/12\\\/Screenshot-from-2022-12-20-23-28-15-1024x404.png\",\"datePublished\":\"2022-12-15T15:11:54+00:00\",\"dateModified\":\"2023-05-09T16:19:46+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/method\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/method\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/method\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/wp-content\\\/uploads\\\/sites\\\/72\\\/2022\\\/12\\\/Screenshot-from-2022-12-20-23-28-15.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/wp-content\\\/uploads\\\/sites\\\/72\\\/2022\\\/12\\\/Screenshot-from-2022-12-20-23-28-15.png\",\"width\":1920,\"height\":758},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/method\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/\",\"name\":\"Sensor Simulation with Dynamic Objects\",\"description\":\"MSCV Capstone Project | Students: Dakshit Agrawal, Jiajie Xu | Advisors: Ioannis Gkioulekas (CMU), Yuning Chai (Cruise), Ashish Shrivastava (Cruise), Siva Karthik Mustikovela (Cruise) | Sponsor: Cruise\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team1\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - Sensor Simulation with Dynamic Objects","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/","og_locale":"en_US","og_type":"article","og_title":"Method - Sensor Simulation with Dynamic Objects","og_description":"Baseline We use the approach mentioned in DynamicNeRF [1] as our baseline. The overall architecture of DynamicNeRF consists of two NeRF models, one for the static scene and the other for the dynamic object. Both of the NeRF models take the following inputs: The static model implements vanilla NeRF, querying color and density given the &hellip; Continue reading \"Method\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/","og_site_name":"Sensor Simulation with Dynamic Objects","article_modified_time":"2023-05-09T16:19:46+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/Screenshot-from-2022-12-20-23-28-15-1024x404.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/","name":"Method - Sensor Simulation with Dynamic Objects","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/Screenshot-from-2022-12-20-23-28-15-1024x404.png","datePublished":"2022-12-15T15:11:54+00:00","dateModified":"2023-05-09T16:19:46+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/Screenshot-from-2022-12-20-23-28-15.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-content\/uploads\/sites\/72\/2022\/12\/Screenshot-from-2022-12-20-23-28-15.png","width":1920,"height":758},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/method\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/","name":"Sensor Simulation with Dynamic Objects","description":"MSCV Capstone Project | Students: Dakshit Agrawal, Jiajie Xu | Advisors: Ioannis Gkioulekas (CMU), Yuning Chai (Cruise), Ashish Shrivastava (Cruise), Siva Karthik Mustikovela (Cruise) | Sponsor: Cruise","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-json\/wp\/v2\/pages\/17","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-json\/wp\/v2\/users\/145"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-json\/wp\/v2\/comments?post=17"}],"version-history":[{"count":11,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-json\/wp\/v2\/pages\/17\/revisions"}],"predecessor-version":[{"id":143,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-json\/wp\/v2\/pages\/17\/revisions\/143"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team1\/wp-json\/wp\/v2\/media?parent=17"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}