{"id":11,"date":"2025-05-06T19:12:30","date_gmt":"2025-05-06T19:12:30","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/?page_id=11"},"modified":"2025-12-11T04:35:44","modified_gmt":"2025-12-11T04:35:44","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/","title":{"rendered":"Method"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\"><strong>Leveraging Geometric Foundation Models (VGGT) for Robotic Manipulation<\/strong><\/h2>\n\n\n\n<p>In our experiments, we adopt VGGT as the representative 3D foundation model. For the manipulation policy, we use 3D Diffusion Policy [1], which operates on point cloud data and is trained via behavior cloning.<\/p>\n\n\n\n<p>We explore two ways of incorporating information from 3D foundation models:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Explicitly: Using the generated points clouds from 3D foundation models<\/h3>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"243\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.39.34\u202fAM-1024x243.png\" alt=\"\" class=\"wp-image-44\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.39.34\u202fAM-1024x243.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.39.34\u202fAM-300x71.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.39.34\u202fAM-768x182.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.39.34\u202fAM-1536x364.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.39.34\u202fAM-2000x476.png 2000w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.39.34\u202fAM.png 2006w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>3D diffusion Policy (DP3) uses ground truth point cloud for simulated data and depth cameras to get point clouds for real world data.<\/li>\n\n\n\n<li>We modify this setup by integrating VGGT, which can utilize multi-view RGB images to generate a point cloud representation of the scene.<\/li>\n\n\n\n<li>The resulting point cloud is subsequently fed into the DP3 point cloud encoder.<\/li>\n\n\n\n<li>The encoder produces a compact 3D representation, which serves as input to the manipulation policy.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Implicitly: Using extracted features from 3D foundation models<\/h3>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"258\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.42.01\u202fAM-1024x258.png\" alt=\"\" class=\"wp-image-45\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.42.01\u202fAM-1024x258.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.42.01\u202fAM-300x76.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.42.01\u202fAM-768x193.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.42.01\u202fAM-1536x387.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/05\/Screenshot-2025-05-07-at-12.42.01\u202fAM.png 2026w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Instead of using a point cloud as input to the DP3 point cloud encoder to generate a compact 3D representation, we use features extracted from VGGT.<\/li>\n\n\n\n<li>We experiment with various bottlenecking strategies to downsample the VGGT features into a compact 3D representation.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>&nbsp;Improving Grasping with Learning-based Shape Completion Networks<\/strong><\/h2>\n\n\n\n<p><br>EconomicGrasp [2] is a system for <strong>6-DOF grasp detection<\/strong> that takes a 3D point cloud of a scene\u2014typically obtained from a single RGB-D frame\u2014and predicts feasible grasp poses in 3D space, including the position, orientation, and grasp quality. The method replaces traditional dense grasp supervision with an <strong>economic supervision strategy<\/strong>, selecting only a compact set of unambiguous grasp labels. A <strong>focal representation module<\/strong> and an <strong>interactive grasp head<\/strong> further refine these candidates, enabling the model to output accurate, high-quality 6-DOF grasps with significantly reduced training and memory cost compared to prior approaches.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"255\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/12\/image-2-1024x255.png\" alt=\"\" class=\"wp-image-133\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/12\/image-2-1024x255.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/12\/image-2-300x75.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/12\/image-2-768x191.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/12\/image-2-1536x383.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/12\/image-2-2048x511.png 2048w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><\/figure>\n\n\n\n<p><br><strong>Our Modification<\/strong><br>In our work, we adapt the EconomicGrasp framework to operate on RaySt3r-completed<strong> <\/strong>point clouds rather than raw single-view RGB-D point clouds. RaySt3r provides a more complete reconstruction of the underlying object by performing zero-shot 3D shape completion, allowing the grasp planner to reason over occluded regions and a more accurate object geometry.<\/p>\n\n\n\n<p>To integrate these richer inputs, we:<br>&#8211; Replace the standard input pipeline with RaySt3r-generated completed point clouds.<br>&#8211; Fine-tune EconomicGrasp on these completed reconstructions so the grasp prediction network learns to exploit RaySt3r\u2019s improved geometric fidelity.<br><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>References<\/strong><\/h2>\n\n\n\n<p>[1] Ze, Yanjie, et al. &#8220;3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations.&#8221; <em>arXiv preprint arXiv:2403.03954<\/em> (2024).<br>[2] Wu, Xiao-Ming, et al. &#8220;An economic framework for 6-dof grasp detection.&#8221; ECCV 2024.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Leveraging Geometric Foundation Models (VGGT) for Robotic Manipulation In our experiments, we adopt VGGT as the representative 3D foundation model. For the manipulation policy, we use 3D Diffusion Policy [1], which operates on point cloud data and is trained via behavior cloning. We explore two ways of incorporating information from 3D foundation models: Explicitly: Using &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Method&#8221;<\/span><\/a><\/p>\n","protected":false},"author":252,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-11","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Method - Improving Robot Manipulation with 3D Vision Models<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - Improving Robot Manipulation with 3D Vision Models\" \/>\n<meta property=\"og:description\" content=\"Leveraging Geometric Foundation Models (VGGT) for Robotic Manipulation In our experiments, we adopt VGGT as the representative 3D foundation model. For the manipulation policy, we use 3D Diffusion Policy [1], which operates on point cloud data and is trained via behavior cloning. We explore two ways of incorporating information from 3D foundation models: Explicitly: Using &hellip; Continue reading &quot;Method&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/\" \/>\n<meta property=\"og:site_name\" content=\"Improving Robot Manipulation with 3D Vision Models\" \/>\n<meta property=\"article:modified_time\" content=\"2025-12-11T04:35:44+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/12\/image-2-1024x255.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/method\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/method\\\/\",\"name\":\"Method - Improving Robot Manipulation with 3D Vision Models\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/method\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/method\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/wp-content\\\/uploads\\\/sites\\\/134\\\/2025\\\/12\\\/image-2-1024x255.png\",\"datePublished\":\"2025-05-06T19:12:30+00:00\",\"dateModified\":\"2025-12-11T04:35:44+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/method\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/method\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/method\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/wp-content\\\/uploads\\\/sites\\\/134\\\/2025\\\/12\\\/image-2.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/wp-content\\\/uploads\\\/sites\\\/134\\\/2025\\\/12\\\/image-2.png\",\"width\":2086,\"height\":520},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/method\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/\",\"name\":\"Improving Robot Manipulation with 3D Vision Models\",\"description\":\"Rajath Aralikatti, Santhoshini Gongidi | Collaborators: Bardenious Duisterhof, Jeffrey Ke | Advisor: Jeffrey Ichnowski\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team14\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - Improving Robot Manipulation with 3D Vision Models","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/","og_locale":"en_US","og_type":"article","og_title":"Method - Improving Robot Manipulation with 3D Vision Models","og_description":"Leveraging Geometric Foundation Models (VGGT) for Robotic Manipulation In our experiments, we adopt VGGT as the representative 3D foundation model. For the manipulation policy, we use 3D Diffusion Policy [1], which operates on point cloud data and is trained via behavior cloning. We explore two ways of incorporating information from 3D foundation models: Explicitly: Using &hellip; Continue reading \"Method\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/","og_site_name":"Improving Robot Manipulation with 3D Vision Models","article_modified_time":"2025-12-11T04:35:44+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/12\/image-2-1024x255.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/","name":"Method - Improving Robot Manipulation with 3D Vision Models","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/12\/image-2-1024x255.png","datePublished":"2025-05-06T19:12:30+00:00","dateModified":"2025-12-11T04:35:44+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/12\/image-2.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-content\/uploads\/sites\/134\/2025\/12\/image-2.png","width":2086,"height":520},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/method\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/","name":"Improving Robot Manipulation with 3D Vision Models","description":"Rajath Aralikatti, Santhoshini Gongidi | Collaborators: Bardenious Duisterhof, Jeffrey Ke | Advisor: Jeffrey Ichnowski","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-json\/wp\/v2\/pages\/11","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-json\/wp\/v2\/users\/252"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-json\/wp\/v2\/comments?post=11"}],"version-history":[{"count":15,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-json\/wp\/v2\/pages\/11\/revisions"}],"predecessor-version":[{"id":146,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-json\/wp\/v2\/pages\/11\/revisions\/146"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/wp-json\/wp\/v2\/media?parent=11"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}