{"id":102,"date":"2023-05-08T14:07:41","date_gmt":"2023-05-08T14:07:41","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/?page_id=102"},"modified":"2023-12-19T04:02:04","modified_gmt":"2023-12-19T04:02:04","slug":"introduction","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/","title":{"rendered":"Project Summary"},"content":{"rendered":"\n<h3 class=\"wp-block-heading\"><strong>Problem Formulation<\/strong><\/h3>\n\n\n\n<p><strong>During Spring 2023, we focused on human motion generative model with increased diversity.<\/strong><\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"585\" height=\"276\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/intro.png\" alt=\"\" class=\"wp-image-96\" style=\"width:233px;height:110px\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/intro.png 585w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/intro-300x142.png 300w\" sizes=\"auto, (max-width: 585px) 100vw, 585px\" \/><\/figure>\n<\/div>\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"720\" height=\"720\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/introfig-1.gif\" alt=\"\" class=\"wp-image-97\" style=\"width:245px;height:245px\" \/><\/figure>\n<\/div>\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Goal: <\/strong>Text-driven human motion synthesis<\/li>\n\n\n\n<li><strong>Input:<\/strong> Text specifying the human motion&nbsp;<\/li>\n\n\n\n<li><strong>Output: <\/strong>Synthesized 3D human motion matching the input text<\/li>\n<\/ul>\n\n\n\n<p><strong>During Fall 2023, we extended our problem to human motion synthesis with human-object interaction.<\/strong><\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:100%\">\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:66.66%\">\n<figure class=\"wp-block-image size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"880\" height=\"441\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-3.png\" alt=\"\" class=\"wp-image-437\" style=\"width:440px;height:auto\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-3.png 880w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-3-300x150.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-3-768x385.png 768w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:33.33%\">\n<p><img decoding=\"async\" src=\"https:\/\/lh7-us.googleusercontent.com\/m0OnMhBgXLfhjaBX2TeP8gPT-JtkT7sxQg3zLDoNzHZ2RhbzXzKrbxYcAWgW8TMVg6AQxlinhZ7w7W1ne743dkjdZ4T7AS4m6QK-hQLseKEScJfY6gK1CXvODYR_1Dj3mJFvz-fB2MYA40VeSUtCdIBqTg=s2048\" style=\"width: 1300px\"><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Goal: <\/strong>Text-driven human motion synthesis with human-object interaction.<\/li>\n\n\n\n<li><strong>Input:<\/strong> Text specifying the interaction between hand and object, initial body pose, initial object mesh<\/li>\n\n\n\n<li><strong>Output: <\/strong>Synthesized 3D human-object interaction motion matching the input text<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Why Challenging?<\/strong><\/h3>\n\n\n\n<p>Building a text-driven human motion synthesis model is a challenging task due to the high degree of freedom involved in human motion, ambiguity of natural language input in text-to-motion synthesis, and difficulty in achieving natural-looking human motions. <\/p>\n\n\n\n<p>Additionally,  incorporating human-object interaction is particularly challenging because humans engage with objects in a highly diverse and detailed manner. Also, human hands can manipulate objects in intricate ways, requiring precise modeling of dexterous movements and the nuanced physical interactions between the hand and the object.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Our Work Summary<\/strong><\/h3>\n\n\n\n<p><strong>MotionGPT (Spring 2023)<\/strong><\/p>\n\n\n\n<p>We proposes MotionGPT, a doubly text-conditional motion diffusion model, to address the limitations of previous text-based human motion generation methods by utilizing the extensive semantic information available in large language models (LLMs). We also enforce the physical constraint in an end-to-end differentiable manner to synthesize more natural human motion. Our method achieves new state-of-the-art quantitative results in terms of FID and motion diversity metrics. Furthermore, it has strong qualitative performance, producing natural results. <\/p>\n\n\n\n<p><strong>Human-Object Interaction (Fall 2024)<\/strong><\/p>\n\n\n\n<p>We introduce a novel diffusion-based framework to synthesize whole-body human hand-object interactions conditioned on text input. Our framework integrates dynamic contact region prediction, ensuring correct contact between the human hand and object. Quantitative results show that our framework consistently achieves high-quality motion performance in both seen and novel action scenarios. <\/p>\n","protected":false},"excerpt":{"rendered":"<p>Problem Formulation During Spring 2023, we focused on human motion generative model with increased diversity. During Fall 2023, we extended our problem to human motion synthesis with human-object interaction. Why Challenging? Building a text-driven human motion synthesis model is a challenging task due to the high degree of freedom involved in human motion, ambiguity of &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Project Summary&#8221;<\/span><\/a><\/p>\n","protected":false},"author":173,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-102","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Project Summary - Human Motion Synthesis<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Project Summary - Human Motion Synthesis\" \/>\n<meta property=\"og:description\" content=\"Problem Formulation During Spring 2023, we focused on human motion generative model with increased diversity. During Fall 2023, we extended our problem to human motion synthesis with human-object interaction. Why Challenging? Building a text-driven human motion synthesis model is a challenging task due to the high degree of freedom involved in human motion, ambiguity of &hellip; Continue reading &quot;Project Summary&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/\" \/>\n<meta property=\"og:site_name\" content=\"Human Motion Synthesis\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-19T04:02:04+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/intro.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/\",\"name\":\"Project Summary - Human Motion Synthesis\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/wp-content\\\/uploads\\\/sites\\\/88\\\/2023\\\/05\\\/intro.png\",\"datePublished\":\"2023-05-08T14:07:41+00:00\",\"dateModified\":\"2023-12-19T04:02:04+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/wp-content\\\/uploads\\\/sites\\\/88\\\/2023\\\/05\\\/intro.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/wp-content\\\/uploads\\\/sites\\\/88\\\/2023\\\/05\\\/intro.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Project Summary\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/\",\"name\":\"Human Motion Synthesis\",\"description\":\"Chih-Chun Yang, Tianhui Cai | Advisor: Fernando De la Torre\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Project Summary - Human Motion Synthesis","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/","og_locale":"en_US","og_type":"article","og_title":"Project Summary - Human Motion Synthesis","og_description":"Problem Formulation During Spring 2023, we focused on human motion generative model with increased diversity. During Fall 2023, we extended our problem to human motion synthesis with human-object interaction. Why Challenging? Building a text-driven human motion synthesis model is a challenging task due to the high degree of freedom involved in human motion, ambiguity of &hellip; Continue reading \"Project Summary\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/","og_site_name":"Human Motion Synthesis","article_modified_time":"2023-12-19T04:02:04+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/intro.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/","name":"Project Summary - Human Motion Synthesis","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/intro.png","datePublished":"2023-05-08T14:07:41+00:00","dateModified":"2023-12-19T04:02:04+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/intro.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/intro.png"},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/"},{"@type":"ListItem","position":2,"name":"Project Summary"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/","name":"Human Motion Synthesis","description":"Chih-Chun Yang, Tianhui Cai | Advisor: Fernando De la Torre","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/pages\/102","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/users\/173"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/comments?post=102"}],"version-history":[{"count":37,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/pages\/102\/revisions"}],"predecessor-version":[{"id":513,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/pages\/102\/revisions\/513"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/media?parent=102"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}