{"id":53,"date":"2020-05-10T22:09:31","date_gmt":"2020-05-10T22:09:31","guid":{"rendered":"http:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/?p=53"},"modified":"2020-12-16T19:08:25","modified_gmt":"2020-12-16T19:08:25","slug":"project-overview","status":"publish","type":"post","link":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/","title":{"rendered":"Project Overview"},"content":{"rendered":"\n<p>Semantic Facial Image Manipulation is a conditional generation task. The goal is to synthesize a facial image conditioned on both identity and target expression (In our settings, identity information is provided as a 2D image and target expression is provided as a FACS code).  <\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-content\/uploads\/sites\/42\/2020\/05\/1bw3s-dboih.gif\" alt=\"\" class=\"wp-image-65\" width=\"642\" height=\"555\" \/><figcaption><strong>Figure 1<\/strong>. <strong>Facial Action Coding System (FACS)<\/strong> is a standard representation of facial expression in behavior science [2]. It breaks down facial expressions into individual components of muscle movement, called Action Units. Each expression is encoded as a vector. Each number in the vector represents one AU and the magnitude of the number represents its intensity. This way of encoding facial expressions is identity agnostic and interpretable.<\/figcaption><\/figure>\n\n\n\n<p><strong>Social Motivation<\/strong><\/p>\n\n\n\n<p>Fujitsu is interested in using FACS-based expression analysis to analyze viewer\u2019s behavior for advertising and to provide feedbacks for video conferences, remote training, etc. But FACS annotated data is limited in amount and often suffer from skewed distribution. Some expressions and Action Units are much more frequent than others.  Using facial expression manipulation can generate FACS annotated data while controlling the distribution of generated data (i.e. we can generate expressions and Action Units we want).<\/p>\n\n\n\n<p><strong>Technical Motivation<\/strong><\/p>\n\n\n\n<p>2D based methods like GANimation [1] manipulates faces in 2D image space, which often results in artifacts in generated images and failures under extreme head pose and lighting condition.<\/p>\n\n\n\n<p><strong>Proposed Pipeline<\/strong><\/p>\n\n\n\n<p>We proposed an encode-manipulate-decode pipeline mainly consisting of two modules, geometric manipulation module and semantic manipulation module. The geometric manipulation module warps the feature maps of input image guided by estimated 3D geometry. In this way, artifacts are reduced by moving the manipulation from image space to feature space. The 3D geometry introduced here helps handle extreme head pose and lightning condition.  The warped feature maps are then fed to semantic manipulation module which synthesizes information that was not the input image <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"394\" src=\"http:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-content\/uploads\/sites\/42\/2020\/05\/rough_pipeline-1024x394.png\" alt=\"\" class=\"wp-image-64\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-content\/uploads\/sites\/42\/2020\/05\/rough_pipeline-1024x394.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-content\/uploads\/sites\/42\/2020\/05\/rough_pipeline-300x115.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-content\/uploads\/sites\/42\/2020\/05\/rough_pipeline-768x296.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-content\/uploads\/sites\/42\/2020\/05\/rough_pipeline.png 1268w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><figcaption><strong>Figure 2<\/strong>. An illustration of our proposed pipeline. It mainly consists of two parts, geometric manipulation module and semantic manipulation module<\/figcaption><\/figure>\n\n\n\n<p><strong>References<\/strong><\/p>\n\n\n\n<p>[1]Pumarola et al. GANimation: Anatomically-aware Facial Animation from a Single Image, ECCV2018 <br \/>[2]Ekman, Paul, Wallace V. Friesen, and Joseph C. Hager. &#8220;Facial action coding system: The manual on CD ROM.&#8221; <em>A Human Face, Salt Lake City<\/em> (2002): 77-254<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Semantic Facial Image Manipulation is a conditional generation task. The goal is to synthesize a facial image conditioned on both identity and target expression (In our settings, identity information is provided as a 2D image and target expression is provided as a FACS code). Social Motivation Fujitsu is interested in using FACS-based expression analysis to &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Project Overview&#8221;<\/span><\/a><\/p>\n","protected":false},"author":92,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-53","post","type-post","status-publish","format-standard","hentry","category-sponsored-by-fujitsu"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Project Overview - Semantic Facial Image Manipulation using 2D\/3D modalities<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Project Overview - Semantic Facial Image Manipulation using 2D\/3D modalities\" \/>\n<meta property=\"og:description\" content=\"Semantic Facial Image Manipulation is a conditional generation task. The goal is to synthesize a facial image conditioned on both identity and target expression (In our settings, identity information is provided as a 2D image and target expression is provided as a FACS code). Social Motivation Fujitsu is interested in using FACS-based expression analysis to &hellip; Continue reading &quot;Project Overview&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/\" \/>\n<meta property=\"og:site_name\" content=\"Semantic Facial Image Manipulation using 2D\/3D modalities\" \/>\n<meta property=\"article:published_time\" content=\"2020-05-10T22:09:31+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-12-16T19:08:25+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-content\/uploads\/sites\/42\/2020\/05\/1bw3s-dboih.gif\" \/>\n<meta name=\"author\" content=\"dail\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"dail\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/2020\\\/05\\\/10\\\/project-overview\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/2020\\\/05\\\/10\\\/project-overview\\\/\"},\"author\":{\"name\":\"dail\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/#\\\/schema\\\/person\\\/03088c999902f70889a047878fb6ec6d\"},\"headline\":\"Project Overview\",\"datePublished\":\"2020-05-10T22:09:31+00:00\",\"dateModified\":\"2020-12-16T19:08:25+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/2020\\\/05\\\/10\\\/project-overview\\\/\"},\"wordCount\":362,\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/2020\\\/05\\\/10\\\/project-overview\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/wp-content\\\/uploads\\\/sites\\\/42\\\/2020\\\/05\\\/1bw3s-dboih.gif\",\"articleSection\":[\"Semantic Facial Image Manipulation Using 2D\\\/3D Modalities\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/2020\\\/05\\\/10\\\/project-overview\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/2020\\\/05\\\/10\\\/project-overview\\\/\",\"name\":\"Project Overview - Semantic Facial Image Manipulation using 2D\\\/3D modalities\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/2020\\\/05\\\/10\\\/project-overview\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/2020\\\/05\\\/10\\\/project-overview\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/wp-content\\\/uploads\\\/sites\\\/42\\\/2020\\\/05\\\/1bw3s-dboih.gif\",\"datePublished\":\"2020-05-10T22:09:31+00:00\",\"dateModified\":\"2020-12-16T19:08:25+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/#\\\/schema\\\/person\\\/03088c999902f70889a047878fb6ec6d\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/2020\\\/05\\\/10\\\/project-overview\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/2020\\\/05\\\/10\\\/project-overview\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/2020\\\/05\\\/10\\\/project-overview\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/wp-content\\\/uploads\\\/sites\\\/42\\\/2020\\\/05\\\/1bw3s-dboih.gif\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/wp-content\\\/uploads\\\/sites\\\/42\\\/2020\\\/05\\\/1bw3s-dboih.gif\",\"width\":500,\"height\":432},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/2020\\\/05\\\/10\\\/project-overview\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Project Overview\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/\",\"name\":\"Semantic Facial Image Manipulation using 2D\\\/3D modalities\",\"description\":\"Sponsored by Fujitsu | Advised by Laszlo A. Jeni \",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/#\\\/schema\\\/person\\\/03088c999902f70889a047878fb6ec6d\",\"name\":\"dail\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/63b83c8755a0dd88752195428a595fd1d6112b5d263417ca17d75e88333f03ea?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/63b83c8755a0dd88752195428a595fd1d6112b5d263417ca17d75e88333f03ea?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/63b83c8755a0dd88752195428a595fd1d6112b5d263417ca17d75e88333f03ea?s=96&d=mm&r=g\",\"caption\":\"dail\"},\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teaml\\\/author\\\/dail\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Project Overview - Semantic Facial Image Manipulation using 2D\/3D modalities","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/","og_locale":"en_US","og_type":"article","og_title":"Project Overview - Semantic Facial Image Manipulation using 2D\/3D modalities","og_description":"Semantic Facial Image Manipulation is a conditional generation task. The goal is to synthesize a facial image conditioned on both identity and target expression (In our settings, identity information is provided as a 2D image and target expression is provided as a FACS code). Social Motivation Fujitsu is interested in using FACS-based expression analysis to &hellip; Continue reading \"Project Overview\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/","og_site_name":"Semantic Facial Image Manipulation using 2D\/3D modalities","article_published_time":"2020-05-10T22:09:31+00:00","article_modified_time":"2020-12-16T19:08:25+00:00","og_image":[{"url":"http:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-content\/uploads\/sites\/42\/2020\/05\/1bw3s-dboih.gif","type":"","width":"","height":""}],"author":"dail","twitter_card":"summary_large_image","twitter_misc":{"Written by":"dail","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/#article","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/"},"author":{"name":"dail","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/#\/schema\/person\/03088c999902f70889a047878fb6ec6d"},"headline":"Project Overview","datePublished":"2020-05-10T22:09:31+00:00","dateModified":"2020-12-16T19:08:25+00:00","mainEntityOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/"},"wordCount":362,"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/#primaryimage"},"thumbnailUrl":"http:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-content\/uploads\/sites\/42\/2020\/05\/1bw3s-dboih.gif","articleSection":["Semantic Facial Image Manipulation Using 2D\/3D Modalities"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/","name":"Project Overview - Semantic Facial Image Manipulation using 2D\/3D modalities","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/#primaryimage"},"thumbnailUrl":"http:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-content\/uploads\/sites\/42\/2020\/05\/1bw3s-dboih.gif","datePublished":"2020-05-10T22:09:31+00:00","dateModified":"2020-12-16T19:08:25+00:00","author":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/#\/schema\/person\/03088c999902f70889a047878fb6ec6d"},"breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-content\/uploads\/sites\/42\/2020\/05\/1bw3s-dboih.gif","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-content\/uploads\/sites\/42\/2020\/05\/1bw3s-dboih.gif","width":500,"height":432},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/2020\/05\/10\/project-overview\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/"},{"@type":"ListItem","position":2,"name":"Project Overview"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/","name":"Semantic Facial Image Manipulation using 2D\/3D modalities","description":"Sponsored by Fujitsu | Advised by Laszlo A. Jeni ","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/#\/schema\/person\/03088c999902f70889a047878fb6ec6d","name":"dail","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/63b83c8755a0dd88752195428a595fd1d6112b5d263417ca17d75e88333f03ea?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/63b83c8755a0dd88752195428a595fd1d6112b5d263417ca17d75e88333f03ea?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/63b83c8755a0dd88752195428a595fd1d6112b5d263417ca17d75e88333f03ea?s=96&d=mm&r=g","caption":"dail"},"url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/author\/dail\/"}]}},"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-json\/wp\/v2\/posts\/53","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-json\/wp\/v2\/users\/92"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-json\/wp\/v2\/comments?post=53"}],"version-history":[{"count":4,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-json\/wp\/v2\/posts\/53\/revisions"}],"predecessor-version":[{"id":83,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-json\/wp\/v2\/posts\/53\/revisions\/83"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-json\/wp\/v2\/media?parent=53"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-json\/wp\/v2\/categories?post=53"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teaml\/wp-json\/wp\/v2\/tags?post=53"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}