{"id":7,"date":"2023-05-02T04:38:13","date_gmt":"2023-05-02T04:38:13","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/?page_id=7"},"modified":"2023-12-13T10:54:15","modified_gmt":"2023-12-13T10:54:15","slug":"problem-formulation","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/","title":{"rendered":"Problem Formulation"},"content":{"rendered":"\n<p>Given a large unlabelled set of multi-view data (multiple cameras from diverse viewpoints), the goal is to improve the label efficiency of a human part segmentation model by leveraging its multi-view capabilities on the unlabelled set using active learning.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><strong>                                RELATED WORK<\/strong><\/pre>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<p><strong>HUMAN PART SEGMENTATION<\/strong><\/p>\n\n\n\n<p>Human part segmentation is a crucial task in computer vision that identifies and outlines body parts in images or videos. Multi-view data can help <strong>disambiguate occluded regions <\/strong>and improve segmentation quality. <strong>CDGNet<\/strong>[1] is a state-of-the-art method that simplifies the problem using horizontal and vertical class distribution labels as supervision signals.  The authors observe that the <strong>human body has hierarchically structured parts, and each body part has its unique position distribution<\/strong>. We used CDGNet for our baseline as it provides a robust pipeline and focuses on directional features for human part segmentation, which is helpful for unusual human poses.<\/p>\n<\/div>\n<\/div>\n\n\n\n<p><strong>ACTIVE LEARNING<\/strong><\/p>\n\n\n\n<p><strong>Active Learning (AL)<\/strong> reduces labeled data required for training by intelligently selecting informative samples for annotation. AL iteratively selects instances from unlabeled data and queries the oracle for their labels. Various query strategies measure the informativeness of samples based on the <strong>model&#8217;s uncertainty, disagreement among models, or expected impact<\/strong> on the model&#8217;s parameters.  <strong>ViewAL<\/strong>[2] introduces an innovative active learning (AL) framework designed explicitly for multi-view data. ViewAL utilizes an AL strategy that selects the most informative samples from the multi-view data to be labeled based on <strong>view consistency\/entropy<\/strong> although they operate on outdoor scene data and are not directly comparable. <\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">                                 <strong>Dataset<\/strong><\/pre>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<p>Unfortunately, <strong>no direct large-scale multi-view dataset<\/strong> with consistent and strong part segmentation labels exists. Humans3.6 has some weakly labeled part segmentation labels. Thus, we create our synthetic dataset based on <strong>SURREAL<\/strong>[3] (Synthetic hUmans foR REAL tasks) data.  The primary advantage of SURREAL is that it provides a massive number of annotated samples with precise ground truth information, eliminating the need for manual annotation, which can be both time-consuming and error-prone. We used SURREAL<strong> to generate ~2.5M samples out of which we used<\/strong> 80k samples, 66k for training, 7k for validation, and 7k for final testing. This was done due to computational constraints and to accelerate iterations. To sample diverse poses, we check the 3D keypoint distance of the meshes in a video stream and only add frames if the difference exceeds a threshold.  We provide the <strong>RGB image, part labels, and pixel-wise depth map<\/strong>. Below is an example of an action from our dataset and six views from the same timestamp.<\/p>\n\n\n\n<figure class=\"wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-1 is-layout-flex wp-block-gallery-is-layout-flex\">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"644\" data-id=\"143\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-07-at-12.09.44-AM-1024x644.png\" alt=\"\" class=\"wp-image-143\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-07-at-12.09.44-AM-1024x644.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-07-at-12.09.44-AM-300x189.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-07-at-12.09.44-AM-768x483.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-07-at-12.09.44-AM.png 1530w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/figure>\n<\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>References:<\/strong><\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Kunliang Liu, Ouk Choi, Jianming Wang, Wonjun Hwang: CDGNet: Class Distribution Guided Network for Human Parsing<\/li>\n\n\n\n<li>Yawar Siddiqui, Julien Valentin, Matthias Nie\u00dfner: ViewAL: Active Learning with Viewpoint Entropy for Semantic Segmentation<\/li>\n\n\n\n<li>G\u00fcl Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, Cordelia Schmid: Learning from Synthetic Humans<\/li>\n<\/ol>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Given a large unlabelled set of multi-view data (multiple cameras from diverse viewpoints), the goal is to improve the label efficiency of a human part segmentation model by leveraging its multi-view capabilities on the unlabelled set using active learning. RELATED WORK HUMAN PART SEGMENTATION Human part segmentation is a crucial task in computer vision that &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Problem Formulation&#8221;<\/span><\/a><\/p>\n","protected":false},"author":161,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-7","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Problem Formulation - Hard Example Mining for Multi-view Human Part Segmentation<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Problem Formulation - Hard Example Mining for Multi-view Human Part Segmentation\" \/>\n<meta property=\"og:description\" content=\"Given a large unlabelled set of multi-view data (multiple cameras from diverse viewpoints), the goal is to improve the label efficiency of a human part segmentation model by leveraging its multi-view capabilities on the unlabelled set using active learning. RELATED WORK HUMAN PART SEGMENTATION Human part segmentation is a crucial task in computer vision that &hellip; Continue reading &quot;Problem Formulation&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/\" \/>\n<meta property=\"og:site_name\" content=\"Hard Example Mining for Multi-view Human Part Segmentation\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-13T10:54:15+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-07-at-12.09.44-AM-1024x644.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/problem-formulation\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/problem-formulation\\\/\",\"name\":\"Problem Formulation - Hard Example Mining for Multi-view Human Part Segmentation\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/problem-formulation\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/problem-formulation\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/wp-content\\\/uploads\\\/sites\\\/81\\\/2023\\\/05\\\/Screenshot-2023-05-07-at-12.09.44-AM-1024x644.png\",\"datePublished\":\"2023-05-02T04:38:13+00:00\",\"dateModified\":\"2023-12-13T10:54:15+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/problem-formulation\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/problem-formulation\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/problem-formulation\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/wp-content\\\/uploads\\\/sites\\\/81\\\/2023\\\/05\\\/Screenshot-2023-05-07-at-12.09.44-AM-1024x644.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/wp-content\\\/uploads\\\/sites\\\/81\\\/2023\\\/05\\\/Screenshot-2023-05-07-at-12.09.44-AM-1024x644.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/problem-formulation\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Problem Formulation\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/\",\"name\":\"Hard Example Mining for Multi-view Human Part Segmentation\",\"description\":\"Students: Ashwin Vaswani, Minh Tran | Advisors: He Wen (Meta), Yuan Dong (Meta), Deepak Pathak (CMU) | Sponsor: Meta\",\"publisher\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/#organization\",\"name\":\"Hard Example Mining for Multi-view Human Part Segmentation\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/wp-content\\\/uploads\\\/sites\\\/81\\\/2023\\\/05\\\/meta.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/wp-content\\\/uploads\\\/sites\\\/81\\\/2023\\\/05\\\/meta.png\",\"width\":225,\"height\":225,\"caption\":\"Hard Example Mining for Multi-view Human Part Segmentation\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Problem Formulation - Hard Example Mining for Multi-view Human Part Segmentation","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/","og_locale":"en_US","og_type":"article","og_title":"Problem Formulation - Hard Example Mining for Multi-view Human Part Segmentation","og_description":"Given a large unlabelled set of multi-view data (multiple cameras from diverse viewpoints), the goal is to improve the label efficiency of a human part segmentation model by leveraging its multi-view capabilities on the unlabelled set using active learning. RELATED WORK HUMAN PART SEGMENTATION Human part segmentation is a crucial task in computer vision that &hellip; Continue reading \"Problem Formulation\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/","og_site_name":"Hard Example Mining for Multi-view Human Part Segmentation","article_modified_time":"2023-12-13T10:54:15+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-07-at-12.09.44-AM-1024x644.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/","name":"Problem Formulation - Hard Example Mining for Multi-view Human Part Segmentation","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-07-at-12.09.44-AM-1024x644.png","datePublished":"2023-05-02T04:38:13+00:00","dateModified":"2023-12-13T10:54:15+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-07-at-12.09.44-AM-1024x644.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-07-at-12.09.44-AM-1024x644.png"},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/problem-formulation\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/"},{"@type":"ListItem","position":2,"name":"Problem Formulation"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/","name":"Hard Example Mining for Multi-view Human Part Segmentation","description":"Students: Ashwin Vaswani, Minh Tran | Advisors: He Wen (Meta), Yuan Dong (Meta), Deepak Pathak (CMU) | Sponsor: Meta","publisher":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/#organization","name":"Hard Example Mining for Multi-view Human Part Segmentation","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/#\/schema\/logo\/image\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/meta.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/meta.png","width":225,"height":225,"caption":"Hard Example Mining for Multi-view Human Part Segmentation"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/pages\/7","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/users\/161"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/comments?post=7"}],"version-history":[{"count":19,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/pages\/7\/revisions"}],"predecessor-version":[{"id":199,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/pages\/7\/revisions\/199"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/media?parent=7"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}