{"id":90,"date":"2023-05-06T18:31:05","date_gmt":"2023-05-06T18:31:05","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/?page_id=90"},"modified":"2023-12-13T10:37:57","modified_gmt":"2023-12-13T10:37:57","slug":"methodology","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/","title":{"rendered":"Methodology"},"content":{"rendered":"\n

Stage 1:<\/strong><\/h2>\n\n\n\n

We start with an initial set of labeled multi-view data. Precisely, 1500 images<\/strong> (150 actions with ten views each) were randomly sampled. This corresponds to ~15%<\/strong> of the total training data. We save this set for all future experiments to maintain consistency. <\/p>\n\n\n\n

\n
\"\"<\/figure>\n<\/figure>\n\n\n\n

Stage 2:<\/strong><\/h2>\n\n\n\n

In this stage, we train a body part segmentation model. We adapt the architecture from CDGNet, the current state-of-the-art model for this task. We replace the ResNet101 architecture with a ResNet50<\/strong> backbone to improve training\/inference time due to multiple iterations of active learning. We downsample the images to a resolution of 360×640<\/strong> for the same reason. We do not supervise the model with multi-view training but pass every image independently<\/strong>.<\/p>\n\n\n\n

\n
\"\"<\/figure>\n<\/figure>\n\n\n\n

Stage 3:<\/strong><\/h2>\n\n\n\n

We save the trained model from the method above and run inference on the multi-view data. We then check for multi-view consistency and diversity of poses according to the following strategy: For every view, we cross project all of the other n-1 views predicted segmentation scores on the given view to compute average segmentation scores. This is then used to compute multi-view uncertainty and we then used entropy\/uncertainty based active learning methods on this averaged out score. We can use mesh rendering to project the segmentation features onto the mesh (if available) or point rendering to project the segmentation features into a common 3D space using depth. These are then used to sample hard examples for the next stage of active learning.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Stage 4:<\/strong><\/h2>\n\n\n\n

To use the model’s notion of easy and hard, we plan to use the hard examples (500 i.e. 5% of available data) with higher weights<\/strong> in this next iteration and easy examples as pseudo labels<\/strong> in the next iteration of training. This is motivated by impressive results from Link<\/a><\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

Stage 1: We start with an initial set of labeled multi-view data. Precisely, 1500 images (150 actions with ten views each) were randomly sampled. This corresponds to ~15% of the total training data. We save this set for all future experiments to maintain consistency. Stage 2: In this stage, we train a body part segmentation … <\/p>\n

Continue reading “Methodology”<\/span><\/a><\/p>\n","protected":false},"author":162,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-90","page","type-page","status-publish","hentry"],"yoast_head":"\nMethodology - Hard Example Mining for Multi-view Human Part Segmentation<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Methodology - Hard Example Mining for Multi-view Human Part Segmentation\" \/>\n<meta property=\"og:description\" content=\"Stage 1: We start with an initial set of labeled multi-view data. Precisely, 1500 images (150 actions with ten views each) were randomly sampled. This corresponds to ~15% of the total training data. We save this set for all future experiments to maintain consistency. Stage 2: In this stage, we train a body part segmentation … Continue reading "Methodology"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/\" \/>\n<meta property=\"og:site_name\" content=\"Hard Example Mining for Multi-view Human Part Segmentation\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-13T10:37:57+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-06-at-10.45.39-PM-1.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/methodology\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/methodology\\\/\",\"name\":\"Methodology - Hard Example Mining for Multi-view Human Part Segmentation\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/methodology\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/methodology\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/wp-content\\\/uploads\\\/sites\\\/81\\\/2023\\\/05\\\/Screenshot-2023-05-06-at-10.45.39-PM-1.png\",\"datePublished\":\"2023-05-06T18:31:05+00:00\",\"dateModified\":\"2023-12-13T10:37:57+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/methodology\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/methodology\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/methodology\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/wp-content\\\/uploads\\\/sites\\\/81\\\/2023\\\/05\\\/Screenshot-2023-05-06-at-10.45.39-PM-1.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/wp-content\\\/uploads\\\/sites\\\/81\\\/2023\\\/05\\\/Screenshot-2023-05-06-at-10.45.39-PM-1.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/methodology\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Methodology\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/\",\"name\":\"Hard Example Mining for Multi-view Human Part Segmentation\",\"description\":\"Students: Ashwin Vaswani, Minh Tran | Advisors: He Wen (Meta), Yuan Dong (Meta), Deepak Pathak (CMU) | Sponsor: Meta\",\"publisher\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/#organization\",\"name\":\"Hard Example Mining for Multi-view Human Part Segmentation\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/wp-content\\\/uploads\\\/sites\\\/81\\\/2023\\\/05\\\/meta.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/wp-content\\\/uploads\\\/sites\\\/81\\\/2023\\\/05\\\/meta.png\",\"width\":225,\"height\":225,\"caption\":\"Hard Example Mining for Multi-view Human Part Segmentation\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team4\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Methodology - Hard Example Mining for Multi-view Human Part Segmentation","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/","og_locale":"en_US","og_type":"article","og_title":"Methodology - Hard Example Mining for Multi-view Human Part Segmentation","og_description":"Stage 1: We start with an initial set of labeled multi-view data. Precisely, 1500 images (150 actions with ten views each) were randomly sampled. This corresponds to ~15% of the total training data. We save this set for all future experiments to maintain consistency. Stage 2: In this stage, we train a body part segmentation … Continue reading \"Methodology\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/","og_site_name":"Hard Example Mining for Multi-view Human Part Segmentation","article_modified_time":"2023-12-13T10:37:57+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-06-at-10.45.39-PM-1.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/","name":"Methodology - Hard Example Mining for Multi-view Human Part Segmentation","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-06-at-10.45.39-PM-1.png","datePublished":"2023-05-06T18:31:05+00:00","dateModified":"2023-12-13T10:37:57+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-06-at-10.45.39-PM-1.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/Screenshot-2023-05-06-at-10.45.39-PM-1.png"},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/methodology\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/"},{"@type":"ListItem","position":2,"name":"Methodology"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/","name":"Hard Example Mining for Multi-view Human Part Segmentation","description":"Students: Ashwin Vaswani, Minh Tran | Advisors: He Wen (Meta), Yuan Dong (Meta), Deepak Pathak (CMU) | Sponsor: Meta","publisher":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/#organization","name":"Hard Example Mining for Multi-view Human Part Segmentation","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/#\/schema\/logo\/image\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/meta.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-content\/uploads\/sites\/81\/2023\/05\/meta.png","width":225,"height":225,"caption":"Hard Example Mining for Multi-view Human Part Segmentation"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/pages\/90","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/users\/162"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/comments?post=90"}],"version-history":[{"count":9,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/pages\/90\/revisions"}],"predecessor-version":[{"id":192,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/pages\/90\/revisions\/192"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team4\/wp-json\/wp\/v2\/media?parent=90"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}