{"id":400,"date":"2023-12-17T20:16:49","date_gmt":"2023-12-17T20:16:49","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/?page_id=400"},"modified":"2023-12-19T02:15:24","modified_gmt":"2023-12-19T02:15:24","slug":"motiongpt-spring-2023","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/","title":{"rendered":"MotionGPT (Spring 2023)"},"content":{"rendered":"\n<p>We proposes a new approach, called MotionGPT, to address the limitations of previous text-based human motion generation methods by utilizing the extensive semantic information available in large language models (LLMs).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Framework of MotionGPT<\/strong><\/h3>\n\n\n\n<p>We first pretrain a doubly text-conditional motion diffusion model on both high-level (such as &#8220;greeting a friend&#8221;) and low-level (such as \u201chug&#8221; or &#8220;wave hand&#8221;) ground truth text data. Then during inference, we improve motion diversity and alignment with the training set, by zero-shot prompting GPT-3 for additional &#8220;low-level&#8221; details.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"602\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-9-1024x602.png\" alt=\"\" class=\"wp-image-487\" style=\"width:661px;height:auto\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-9-1024x602.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-9-300x176.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-9-768x451.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-9-1536x902.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-9.png 1942w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><figcaption class=\"wp-element-caption\">Framework of MotionGPT<\/figcaption><\/figure>\n\n\n\n<p>Our denoising network architecture is based on MDM and employs a transformer with four inputs: the training motion sample x of variable length, the encoding of the denoising step index t, the positional embedding of the temporal ordering of motion frames, and the conditioning vector x. These inputs are linearly projected into a common 512-dimension space and added together to form the input of the transformer encoder.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>GPT-3 Prompting<\/strong><\/h3>\n\n\n\n<p>During inference, we utilize zero-shot prompting for the GPT-3 LLM to generate \u201clow-level\u201d descriptions for each \u201chigh-level\u201d test query. <\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"478\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-10-1024x478.png\" alt=\"\" class=\"wp-image-491\" style=\"width:674px;height:auto\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-10-1024x478.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-10-300x140.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-10-768x358.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-10-1536x717.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-10-2048x956.png 2048w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><figcaption class=\"wp-element-caption\">Overview of our GPT prompting method<\/figcaption><\/figure>\n\n\n\n<p>The Sentence-T5 encoder is used to map the \u201chigh-level\u201d test query and the \u201chigh-level\u201d texts of the training set to the embedding space. We retrieve the top-k \u201chigh-level\u201d training texts that are closest to the test query in terms of cosine distance, as well as their respective \u201clow-level\u201d counterparts. The GPT-3 prompt is formed by concatenating the retrieved \u201chigh-\u201d and \u201clow-level\u201d training texts and the test query.<\/p>\n\n\n\n<p>This approach has two benefits:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>it increases the diversity of text prompts, resulting in a wider range of generated motions.<\/li>\n\n\n\n<li>it allows for better generalization by achieving greater similarity to the training data distribution. This is done by breaking down hard \u201chigh-level\u201d test set descriptions into similar \u201clow-level\u201d actions found in the training set.<\/li>\n<\/ol>\n\n\n\n<p><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Physics-based Regulator<\/strong><\/h3>\n\n\n\n<p>To make the synthesized human motion more natural, we enforce the following physical constraint as extra learning objective. Here we use the same ones as MDM, metioned in Related Work Section.<\/p>\n\n\n\n<p><strong>Foot Contact Constraint: <\/strong><\/p>\n\n\n\n<p><strong>Goal:<\/strong> Make the foot contact the ground appropriately.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"957\" height=\"132\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/\u672a\u547d\u540d\u7c21\u5831-4.png\" alt=\"\" class=\"wp-image-361\" style=\"width:299px;height:40px\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/\u672a\u547d\u540d\u7c21\u5831-4.png 957w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/\u672a\u547d\u540d\u7c21\u5831-4-300x41.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/\u672a\u547d\u540d\u7c21\u5831-4-768x106.png 768w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><\/figure>\n\n\n\n<p>c is the ground truth mask indicating whether the feet contact the ground. If there is contact, we should stop the movement of the feet. <\/p>\n\n\n\n<p><strong>Temporal Constraint:<\/strong> <\/p>\n\n\n\n<p><strong>Goal:<\/strong> Make the human motion smooth.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"80\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/\u672a\u547d\u540d\u7c21\u5831-5-1024x80.png\" alt=\"\" class=\"wp-image-363\" style=\"width:468px;height:36px\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/\u672a\u547d\u540d\u7c21\u5831-5-1024x80.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/\u672a\u547d\u540d\u7c21\u5831-5-300x23.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/\u672a\u547d\u540d\u7c21\u5831-5-768x60.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/\u672a\u547d\u540d\u7c21\u5831-5-1536x119.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/05\/\u672a\u547d\u540d\u7c21\u5831-5.png 1697w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><\/figure>\n\n\n\n<p>This constraint enforce the velocity of the synthesized motion should be the same as the ground truth velocity.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We proposes a new approach, called MotionGPT, to address the limitations of previous text-based human motion generation methods by utilizing the extensive semantic information available in large language models (LLMs). Framework of MotionGPT We first pretrain a doubly text-conditional motion diffusion model on both high-level (such as &#8220;greeting a friend&#8221;) and low-level (such as \u201chug&#8221; &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;MotionGPT (Spring 2023)&#8221;<\/span><\/a><\/p>\n","protected":false},"author":173,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-400","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>MotionGPT (Spring 2023) - Human Motion Synthesis<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"MotionGPT (Spring 2023) - Human Motion Synthesis\" \/>\n<meta property=\"og:description\" content=\"We proposes a new approach, called MotionGPT, to address the limitations of previous text-based human motion generation methods by utilizing the extensive semantic information available in large language models (LLMs). Framework of MotionGPT We first pretrain a doubly text-conditional motion diffusion model on both high-level (such as &#8220;greeting a friend&#8221;) and low-level (such as \u201chug&#8221; &hellip; Continue reading &quot;MotionGPT (Spring 2023)&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/\" \/>\n<meta property=\"og:site_name\" content=\"Human Motion Synthesis\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-19T02:15:24+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-9-1024x602.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/motiongpt-spring-2023\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/motiongpt-spring-2023\\\/\",\"name\":\"MotionGPT (Spring 2023) - Human Motion Synthesis\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/motiongpt-spring-2023\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/motiongpt-spring-2023\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/wp-content\\\/uploads\\\/sites\\\/88\\\/2023\\\/12\\\/image-9-1024x602.png\",\"datePublished\":\"2023-12-17T20:16:49+00:00\",\"dateModified\":\"2023-12-19T02:15:24+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/motiongpt-spring-2023\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/motiongpt-spring-2023\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/motiongpt-spring-2023\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/wp-content\\\/uploads\\\/sites\\\/88\\\/2023\\\/12\\\/image-9.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/wp-content\\\/uploads\\\/sites\\\/88\\\/2023\\\/12\\\/image-9.png\",\"width\":1942,\"height\":1141},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/motiongpt-spring-2023\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"MotionGPT (Spring 2023)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/\",\"name\":\"Human Motion Synthesis\",\"description\":\"Chih-Chun Yang, Tianhui Cai | Advisor: Fernando De la Torre\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team11\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"MotionGPT (Spring 2023) - Human Motion Synthesis","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/","og_locale":"en_US","og_type":"article","og_title":"MotionGPT (Spring 2023) - Human Motion Synthesis","og_description":"We proposes a new approach, called MotionGPT, to address the limitations of previous text-based human motion generation methods by utilizing the extensive semantic information available in large language models (LLMs). Framework of MotionGPT We first pretrain a doubly text-conditional motion diffusion model on both high-level (such as &#8220;greeting a friend&#8221;) and low-level (such as \u201chug&#8221; &hellip; Continue reading \"MotionGPT (Spring 2023)\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/","og_site_name":"Human Motion Synthesis","article_modified_time":"2023-12-19T02:15:24+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-9-1024x602.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/","name":"MotionGPT (Spring 2023) - Human Motion Synthesis","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-9-1024x602.png","datePublished":"2023-12-17T20:16:49+00:00","dateModified":"2023-12-19T02:15:24+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-9.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-content\/uploads\/sites\/88\/2023\/12\/image-9.png","width":1942,"height":1141},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/motiongpt-spring-2023\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/"},{"@type":"ListItem","position":2,"name":"MotionGPT (Spring 2023)"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/","name":"Human Motion Synthesis","description":"Chih-Chun Yang, Tianhui Cai | Advisor: Fernando De la Torre","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/pages\/400","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/users\/173"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/comments?post=400"}],"version-history":[{"count":5,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/pages\/400\/revisions"}],"predecessor-version":[{"id":498,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/pages\/400\/revisions\/498"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team11\/wp-json\/wp\/v2\/media?parent=400"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}