{"id":124,"date":"2023-12-18T21:42:16","date_gmt":"2023-12-18T21:42:16","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/?p=124"},"modified":"2023-12-18T21:52:19","modified_gmt":"2023-12-18T21:52:19","slug":"home","status":"publish","type":"post","link":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/","title":{"rendered":"Home"},"content":{"rendered":"\n<h1 class=\"wp-block-heading has-text-align-center\" style=\"font-style:normal;font-weight:600\">Abstract<\/h1>\n\n\n\n<p>This project aims at understanding kitchen in 3D to enable robot learning by watching video demonstrations. Humans are really good at watching demonstration videos and imitating given instructions such as cooking recipes but robots haven&#8217;t achieved that level yet where they can understand the 3D scene around them just by looking at 2D videos. Robots with such abilities can significantly impact the automation industry, not limited to kitchens or restaurants. We have found that the main bottleneck in accomplishing this task is training robots to learn generalizable manipulations skills because of variation in similar objects around us and lack of real-world datasets. Our goal in this project is to develop a data capture system that can yield a variety of human-level and scene-level annotations without relying on an expert for labeling that can further act as priors for robot learning algorithms.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"457\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/abstract.png\" alt=\"\" class=\"wp-image-125\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/abstract.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/abstract-300x134.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/abstract-768x343.png 768w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/figure>\n\n\n\n<h1 class=\"wp-block-heading has-text-align-center\" style=\"font-style:normal;font-weight:600\">Research Objectives<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Develop a multi-camera 3D kitchen capture system<\/li>\n\n\n\n<li>Investigate algorithms for human-level annotations like human body pose and hand pose<\/li>\n\n\n\n<li>Develop an unsupervised 3D scene segmentation algorithm leveraging foundation models<\/li>\n<\/ul>\n\n\n\n<h1 class=\"wp-block-heading has-text-align-center\" style=\"font-style:normal;font-weight:600\">Motivation<\/h1>\n\n\n\n<p>Among related datasets on the Internet, we found GMU Kitchen and ScanNet the most relevant to our task of scene understanding; however, none of them have 3D information, human activity, and kitchen environments simultaneously. Therefore, we are building a pipeline to collect our own dataset in a kitchen environment that includes both 3D information and human activities.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td>Dataset<\/td><td>3D Information<\/td><td>Human Activity<\/td><td>Kitchen<\/td><\/tr><tr><td>Epic Kitchen<\/td><td>\u2718<\/td><td>\u2714<\/td><td>\u2714<\/td><\/tr><tr><td>GMU Kitchen<\/td><td>\u2714<\/td><td>\u2718<\/td><td>\u2714<\/td><\/tr><tr><td>MSR Action3D<\/td><td>\u2714<\/td><td>\u2714<\/td><td>\u2718<\/td><\/tr><tr><td>ScanNet<\/td><td>\u2714<\/td><td>\u2718<\/td><td>\u2718<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">Comparison of Some Existing Datasets<\/figcaption><\/figure>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"315\" height=\"210\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/gmukitchen.png\" alt=\"\" class=\"wp-image-128\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/gmukitchen.png 315w, https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/gmukitchen-300x200.png 300w\" sizes=\"auto, (max-width: 315px) 100vw, 315px\" \/><figcaption class=\"wp-element-caption\">Sample scene from GMU Kitchen<\/figcaption><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"301\" height=\"219\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/scannet.png\" alt=\"\" class=\"wp-image-129\" \/><figcaption class=\"wp-element-caption\">Sample scene from ScanNet<\/figcaption><\/figure>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Abstract This project aims at understanding kitchen in 3D to enable robot learning by watching video demonstrations. Humans are really good at watching demonstration videos and imitating given instructions such as cooking recipes but robots haven&#8217;t achieved that level yet where they can understand the 3D scene around them just by looking at 2D videos. &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Home&#8221;<\/span><\/a><\/p>\n","protected":false},"author":169,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-124","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Home - 3D Kitchen Understanding<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Home - 3D Kitchen Understanding\" \/>\n<meta property=\"og:description\" content=\"Abstract This project aims at understanding kitchen in 3D to enable robot learning by watching video demonstrations. Humans are really good at watching demonstration videos and imitating given instructions such as cooking recipes but robots haven&#8217;t achieved that level yet where they can understand the 3D scene around them just by looking at 2D videos. &hellip; Continue reading &quot;Home&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/\" \/>\n<meta property=\"og:site_name\" content=\"3D Kitchen Understanding\" \/>\n<meta property=\"article:published_time\" content=\"2023-12-18T21:42:16+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-18T21:52:19+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/abstract.png\" \/>\n<meta name=\"author\" content=\"achleshl\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"achleshl\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/2023\\\/12\\\/18\\\/home\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/2023\\\/12\\\/18\\\/home\\\/\"},\"author\":{\"name\":\"achleshl\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/#\\\/schema\\\/person\\\/6365ce88474ebf4013760ad402320da9\"},\"headline\":\"Home\",\"datePublished\":\"2023-12-18T21:42:16+00:00\",\"dateModified\":\"2023-12-18T21:52:19+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/2023\\\/12\\\/18\\\/home\\\/\"},\"wordCount\":261,\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/2023\\\/12\\\/18\\\/home\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/wp-content\\\/uploads\\\/sites\\\/85\\\/2023\\\/12\\\/abstract.png\",\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/2023\\\/12\\\/18\\\/home\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/2023\\\/12\\\/18\\\/home\\\/\",\"name\":\"Home - 3D Kitchen Understanding\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/2023\\\/12\\\/18\\\/home\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/2023\\\/12\\\/18\\\/home\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/wp-content\\\/uploads\\\/sites\\\/85\\\/2023\\\/12\\\/abstract.png\",\"datePublished\":\"2023-12-18T21:42:16+00:00\",\"dateModified\":\"2023-12-18T21:52:19+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/#\\\/schema\\\/person\\\/6365ce88474ebf4013760ad402320da9\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/2023\\\/12\\\/18\\\/home\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/2023\\\/12\\\/18\\\/home\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/2023\\\/12\\\/18\\\/home\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/wp-content\\\/uploads\\\/sites\\\/85\\\/2023\\\/12\\\/abstract.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/wp-content\\\/uploads\\\/sites\\\/85\\\/2023\\\/12\\\/abstract.png\",\"width\":1024,\"height\":457},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/2023\\\/12\\\/18\\\/home\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Home\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/\",\"name\":\"3D Kitchen Understanding\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/#\\\/schema\\\/person\\\/6365ce88474ebf4013760ad402320da9\",\"name\":\"achleshl\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/6687ebf50bf21ae5f186b3d766400f03f98c316465dafc024b52ddad21594407?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/6687ebf50bf21ae5f186b3d766400f03f98c316465dafc024b52ddad21594407?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/6687ebf50bf21ae5f186b3d766400f03f98c316465dafc024b52ddad21594407?s=96&d=mm&r=g\",\"caption\":\"achleshl\"},\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team8\\\/author\\\/achleshl\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Home - 3D Kitchen Understanding","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/","og_locale":"en_US","og_type":"article","og_title":"Home - 3D Kitchen Understanding","og_description":"Abstract This project aims at understanding kitchen in 3D to enable robot learning by watching video demonstrations. Humans are really good at watching demonstration videos and imitating given instructions such as cooking recipes but robots haven&#8217;t achieved that level yet where they can understand the 3D scene around them just by looking at 2D videos. &hellip; Continue reading \"Home\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/","og_site_name":"3D Kitchen Understanding","article_published_time":"2023-12-18T21:42:16+00:00","article_modified_time":"2023-12-18T21:52:19+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/abstract.png","type":"","width":"","height":""}],"author":"achleshl","twitter_card":"summary_large_image","twitter_misc":{"Written by":"achleshl","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/#article","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/"},"author":{"name":"achleshl","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/#\/schema\/person\/6365ce88474ebf4013760ad402320da9"},"headline":"Home","datePublished":"2023-12-18T21:42:16+00:00","dateModified":"2023-12-18T21:52:19+00:00","mainEntityOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/"},"wordCount":261,"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/abstract.png","inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/","name":"Home - 3D Kitchen Understanding","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/abstract.png","datePublished":"2023-12-18T21:42:16+00:00","dateModified":"2023-12-18T21:52:19+00:00","author":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/#\/schema\/person\/6365ce88474ebf4013760ad402320da9"},"breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/abstract.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-content\/uploads\/sites\/85\/2023\/12\/abstract.png","width":1024,"height":457},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/2023\/12\/18\/home\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/"},{"@type":"ListItem","position":2,"name":"Home"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/","name":"3D Kitchen Understanding","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/#\/schema\/person\/6365ce88474ebf4013760ad402320da9","name":"achleshl","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/6687ebf50bf21ae5f186b3d766400f03f98c316465dafc024b52ddad21594407?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/6687ebf50bf21ae5f186b3d766400f03f98c316465dafc024b52ddad21594407?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/6687ebf50bf21ae5f186b3d766400f03f98c316465dafc024b52ddad21594407?s=96&d=mm&r=g","caption":"achleshl"},"url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/author\/achleshl\/"}]}},"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-json\/wp\/v2\/posts\/124","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-json\/wp\/v2\/users\/169"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-json\/wp\/v2\/comments?post=124"}],"version-history":[{"count":3,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-json\/wp\/v2\/posts\/124\/revisions"}],"predecessor-version":[{"id":137,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-json\/wp\/v2\/posts\/124\/revisions\/137"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-json\/wp\/v2\/media?parent=124"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-json\/wp\/v2\/categories?post=124"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team8\/wp-json\/wp\/v2\/tags?post=124"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}