{"id":224,"date":"2024-05-12T23:29:03","date_gmt":"2024-05-12T23:29:03","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/?page_id=224"},"modified":"2024-12-14T04:51:56","modified_gmt":"2024-12-14T04:51:56","slug":"methodology-aviral-agrawal","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/","title":{"rendered":"Methodology"},"content":{"rendered":"\n

Problem Statement<\/strong><\/p>\n\n\n

\n
\n
\n

Grasping and manipulating transparent objects poses a significant challenge for robots. Recent work showed neural radiance fields (NeRFs) work well for depth perception in scenes with transparent objects, and these depth maps can be used to grasp transparent objects with high accuracy. NeRF- based depth reconstruction can still struggle with challenging transparent objects and lighting conditions. In this work, we study the performance of Gaussian Splatting (3DGS) for depth perception of transparent objects.<\/p>\n

We propose two methods, Clear-Splatting [43]<\/strong><\/em> and ClearSplatting-2.0<\/strong><\/em>.<\/p>\n

Clear- Splatting: \u2b50\ufe0f Spotlight Presentation at RoboNeRF workshop at ICRA-2024<\/strong><\/em><\/h2>\n<\/div>\n<\/div>\n<\/div>\n

We first train vanilla 3D Gaussian Splatting (3DGS) on our synthetic blender dataset consisting of transparent objects on a table-top setting. In order to train a 3DGS we need a multi-view dataset and the output from the algorithm is a point cloud. This point cloud is an explicit representation that can be very useful for downstream tasks since it is easy to manipulate an explicit representation such as a point cloud.<\/p>\n\n\n

\n
\n
\n
\"\"
Training view<\/figcaption><\/figure>\n<\/div>\n\n\n\n
\n
\"\"
3DGS learnt geometry<\/figcaption><\/figure>\n<\/div>\n<\/div>\n<\/div><\/div>\n\n\n\n

Fig5. 3D Gaussian Splatting base algorithm<\/p>\n\n\n\n

\n
\n
\"\"
Rendered view (RGB)<\/figcaption><\/figure>\n<\/div><\/div>\n\n\n\n
\n
\"\"
Rendered view Depth<\/figcaption><\/figure>\n<\/div><\/div>\n<\/div>\n\n\n\n

Fig6. Outputs from baseline 3D Gaussian Splatting<\/p>\n\n\n

From figures 5 and 6 we can see that the geometry learnt by 3DGS is highly plagued with floaters. These floaters are not only visible in the point cloud but also in the depth rendering in figure 6. Since 3DGS is optimized in RGB domain, hence, the RGB rendering looks fine visually but the flaw in the geometry is brought out in the depth maps.<\/p>\n\n\n

<\/p>\n\n\n\n

Proposed Approach<\/strong><\/p>\n\n\n

\n
\"\"
Fig7. Clear-Splatting approach overview<\/figcaption><\/figure>\n<\/div>\n\n\n

Learning Residual Splats<\/em><\/h3>\n\n\n
\n
\n
\n

Clear-Splatting first initializes Gaussians in random states and optimizes for view reconstruction to find background Splats of the environment without the transparent object present. We then reset the optimizer and introduce a new smaller set of residual Gaussians, and optimize the state of all Gaussians simultaneously. The background Gaussians are not frozen to facilitate learning of visual effects introduced by the added object, such as shadow and reflections.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n

Depth from Gaussian Splatting<\/em><\/h3>\n\n\n
\n
\n
\n

For Clear-Splatting, we adopt an approach inspired by Ichnowski et al. [1] and Luiten et al. [39]. We set the per- pixel depth as the depth of the Gaussian center where the accumulated transmittance of the ray drops below a threshold ‘m<\/em>‘. If a ray does not reach this threshold it is assigned a high default depth. This method of calculating depth avoids perceiving floaters around depth boundaries compared to alpha-blending the depths of the Gaussians. We use m<\/em> = 0.7 in our experiments, obtained empirically.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n

Depth based Gaussian pruning<\/h3>\n\n\n

In addition, we also introduce depth pruning where we prune the Gaussians that are within a threshold d distance in a given view, at a set frequency of training iterations and lasts until Gaussian densification [10] happens to help the remaining Gaussians adjust the learnt features. This is instrumental in dealing with occlusions in the depth domain, resulting primarily from floaters which are sets of unoptimized Gaussians that persist due to textureless background in the training views.<\/p>\n

Figure 8 shows the final learnt geometry represented as point cloud. In comparison to figure 5 we see that the point cloud is much more defined and hence, leads to better depth estimation.\u00a0<\/p>\n\n

\n
\"\"
Fig8. Clear-Splatting learnt geometry<\/figcaption><\/figure>\n<\/div>\n\n

ClearSplatting-2.0: Using imperfect world models for depth distillation<\/strong><\/em><\/h2>\n

A major assumption of Clear-Splatting was the presence of background images in isolation, i.e, without the transparent object in the scene. Learning the background scene was a way to introduce priors which led to a better geometry capture for the transparent objects when added subsequently into the background scene. However, this was also a limitation of the method since we might not always have the background-only images available. To overcome this limitation, we propose the use of ‘world’ models for injecting priors while modeling the 3D scene.<\/p>\n

Since we tackle the task of metric depth estimation, we propose to use monocular depth estimators. However, depth estimators tend to fail for transparent objects. Amongst all the monocular depth estimators that we tested, Depth Anything V2 [42] resulted in the best performance for transparent objects. However, even Depth Anything V2 had its limitations, as shown in figure 9.<\/p>\n\n\n

\n
\n
\n
\n
\"\"
Input RGB image<\/figcaption><\/figure>\n<\/div>\n\n\n\n
\n
\"\"
Depth Anything V2 output<\/figcaption><\/figure>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n

Fig9. Failure case of Depth Anything V2 on transparent object<\/p>\n\n

Even though monocular depth estimators fail on transparent objects, they are excellent tools for distilling world priors while 3D modelling the scene. <\/span>We come up with a novel approach to distill information from them while staying robust to the imperfections. <\/span>We break the training into two stages, 1) using the monocular depth estimator in-distribution, and 2) using it out-of-distribution.<\/span><\/p>\n

Stage 1<\/h3>\n

We train the 3DGS algorithm with depth supervision. The RGB images are fed to the Depth Anything V2 (DAV2) model to obtain an approximation of the ground truth depth. Since the depth obtained from monocular depth estimators have scale shift, we normalize the output of DAV2 and also normalize the rendered depth from 3DGS. We calculate the L1 loss between the normalized values to perform depth supervision for stage 1 training as shown in figure 10.<\/p>\n\n\n

\"\"
Fig10. Stage-1 training for ClearSplatting-2.0<\/figcaption><\/figure>\n\n\n

Stage 2<\/h3>\n

Simply executing stage 1 is not enough since the depth maps obtained from DAV2 are imperfect, as demonstrated in figure 9. Hence, we also perform stage 2 training.<\/p>\n

In stage 2, we operate DAV2 with out-of-distribution data. Instead of feedig RGB training images to DAV2, we render depth from the stage 1 trained 3DGS and feed them to DAV2 to obtain the monocular depth maps, as shown in figure 11.<\/p>\n\n\n

\n
\n
\"\"
RGB training image<\/figcaption><\/figure>\n<\/div>\n\n\n\n
\n
\"\"
3DGS rendered depth<\/figcaption><\/figure>\n<\/div>\n\n\n\n
\n
\"\"
3DGS depth -> DAV2 output<\/figcaption><\/figure>\n<\/div>\n<\/div>\n\n\n

Fig11. Depth Anything V2 output given 3DGS rendered depth as input<\/p>\n\n\n

From figure 11 we see that DAV2 is now able to generate correct depth for the wine glass (upto a scale) in contrast to figure 9. Thus, we now use this output from DAV2 as the ground truth depth and perform the depth supervision similar to stage 1.<\/p>\n\n\n\n

\"\"
Fig12. Stage-2 training for ClearSplatting-2.0<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"

Problem Statement Grasping and manipulating transparent objects poses a significant challenge for robots. Recent work showed neural radiance fields (NeRFs) work well for depth perception in scenes with transparent objects, and these depth maps can be used to grasp transparent objects with high accuracy. NeRF- based depth reconstruction can still struggle with challenging transparent objects … <\/p>\n

Continue reading “Methodology”<\/span><\/a><\/p>\n","protected":false},"author":210,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-224","page","type-page","status-publish","hentry"],"yoast_head":"\nMethodology - MS Computer Vision Capstone | Team 10<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Methodology - MS Computer Vision Capstone | Team 10\" \/>\n<meta property=\"og:description\" content=\"Problem Statement Grasping and manipulating transparent objects poses a significant challenge for robots. Recent work showed neural radiance fields (NeRFs) work well for depth perception in scenes with transparent objects, and these depth maps can be used to grasp transparent objects with high accuracy. NeRF- based depth reconstruction can still struggle with challenging transparent objects … Continue reading "Methodology"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/\" \/>\n<meta property=\"og:site_name\" content=\"MS Computer Vision Capstone | Team 10\" \/>\n<meta property=\"article:modified_time\" content=\"2024-12-14T04:51:56+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-content\/uploads\/sites\/108\/2024\/05\/training_views.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/methodology-aviral-agrawal\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/methodology-aviral-agrawal\\\/\",\"name\":\"Methodology - MS Computer Vision Capstone | Team 10\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/methodology-aviral-agrawal\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/methodology-aviral-agrawal\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/wp-content\\\/uploads\\\/sites\\\/108\\\/2024\\\/05\\\/training_views.png\",\"datePublished\":\"2024-05-12T23:29:03+00:00\",\"dateModified\":\"2024-12-14T04:51:56+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/methodology-aviral-agrawal\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/methodology-aviral-agrawal\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/methodology-aviral-agrawal\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/wp-content\\\/uploads\\\/sites\\\/108\\\/2024\\\/05\\\/training_views.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/wp-content\\\/uploads\\\/sites\\\/108\\\/2024\\\/05\\\/training_views.png\",\"width\":784,\"height\":738},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/methodology-aviral-agrawal\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Methodology\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/\",\"name\":\"Interpretability using Diffusion\",\"description\":\"\",\"alternateName\":\"iud\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team10\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Methodology - MS Computer Vision Capstone | Team 10","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/","og_locale":"en_US","og_type":"article","og_title":"Methodology - MS Computer Vision Capstone | Team 10","og_description":"Problem Statement Grasping and manipulating transparent objects poses a significant challenge for robots. Recent work showed neural radiance fields (NeRFs) work well for depth perception in scenes with transparent objects, and these depth maps can be used to grasp transparent objects with high accuracy. NeRF- based depth reconstruction can still struggle with challenging transparent objects … Continue reading \"Methodology\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/","og_site_name":"MS Computer Vision Capstone | Team 10","article_modified_time":"2024-12-14T04:51:56+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-content\/uploads\/sites\/108\/2024\/05\/training_views.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/","name":"Methodology - MS Computer Vision Capstone | Team 10","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-content\/uploads\/sites\/108\/2024\/05\/training_views.png","datePublished":"2024-05-12T23:29:03+00:00","dateModified":"2024-12-14T04:51:56+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-content\/uploads\/sites\/108\/2024\/05\/training_views.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-content\/uploads\/sites\/108\/2024\/05\/training_views.png","width":784,"height":738},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/methodology-aviral-agrawal\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/"},{"@type":"ListItem","position":2,"name":"Methodology"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/","name":"Interpretability using Diffusion","description":"","alternateName":"iud","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-json\/wp\/v2\/pages\/224","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-json\/wp\/v2\/users\/210"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-json\/wp\/v2\/comments?post=224"}],"version-history":[{"count":7,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-json\/wp\/v2\/pages\/224\/revisions"}],"predecessor-version":[{"id":344,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-json\/wp\/v2\/pages\/224\/revisions\/344"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team10\/wp-json\/wp\/v2\/media?parent=224"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}