{"id":311,"date":"2022-12-21T00:44:24","date_gmt":"2022-12-21T00:44:24","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/?page_id=311"},"modified":"2022-12-21T00:46:29","modified_gmt":"2022-12-21T00:46:29","slug":"spring-2022","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/","title":{"rendered":"Spring 2022"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Datasets<\/h2>\n\n\n\n<p>Some of the datasets we experimented with are as follows:<\/p>\n\n\n\n<p>(I) 2D Keypoints datasets<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>COCO keypoint detection[1]<\/li><li>MPII Human Pose dataset [2]<\/li><\/ul>\n\n\n\n<p>(II) 3D human mesh datasets<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Human3.6M [3]<\/li><li>3D Poses in the Wild [4]<\/li><li>AGORA [5]<\/li><li>Relative Human [6]<\/li><\/ul>\n\n\n\n<p>However, since these datasets aren\u2019t relevant to our target domain, we worked on creating our own Synthetic dataset using GTA 5 similar to the JTA [7] dataset video below:<video controls=\"\" loop=\"\" muted=\"\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/04\/JTA.mp4\"><\/video><\/p>\n\n\n\n<p>Creating our Synthetic dataset gives us a lot of control, in a lot of aspects like lighting conditions (day or night), camera angles, camera location, number of pedestrians and vehicles, etc. Once we have trained our pose estimation model on the synthetic dataset, we will apply domain adaptation techniques to transfer knowledge to estimate pose in real-world traffic intersections.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"341\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/image-11-1024x341.png\" alt=\"\" class=\"wp-image-215\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/image-11-1024x341.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/image-11-300x100.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/image-11-768x256.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/image-11.png 1068w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><figcaption>We get free supervised data from games<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"372\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/img2-1024x372.jpg\" alt=\"\" class=\"wp-image-217\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/img2-1024x372.jpg 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/img2-300x109.jpg 300w, https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/img2-768x279.jpg 768w, https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/img2.jpg 1052w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><figcaption>We chose the dataset since it\u2019s closest to our target domain<\/figcaption><\/figure>\n\n\n\n<p>Sample video from the dataset we are creating:<video controls=\"\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/04\/GT.webm\"><\/video>It took us about 15 minutes to produce 1 video in a single view<\/p>\n\n\n\n<p>However, the challenges with synthetic datasets are<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>It is very time-consuming to build a dataset. For example, it took 15 minutes to obtain a 1-minute single-view labeled video sequence.<\/li><li>Getting data from multiple views requires adding extensive mods to the game.<\/li><li>Domain transfer to real-world scenarios would be a challenge.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Metrics<\/h2>\n\n\n\n<p>To evaluate our pose estimation model, we use the following metrics:<\/p>\n\n\n\n<p>(a) Mean Per Joint Position Error (MPJPE): Mean of per joint position error for all k joints (Typically, k = 16) Calculated after aligning the root joints (typically the pelvis) of the estimated and ground truth 3D pose.<\/p>\n\n\n\n<p>(b) Percentage of Correct Keypoints (PCK): PCK is used as an accuracy metric that measures if the predicted keypoint and the true joint are within a certain distance threshold. The PCK is usually set with respect to the scale of the subject, which is enclosed within the bounding box.<\/p>\n\n\n\n<p>(c) Per Vertex Error (PVE): It is possible to use MPJPE to evaluate all the 6890 predicted vertices of the SMPL model. In this case, the ground truth SMPL parameters need to be provided and the only dataset available for this purpose is the 3DPW. This type of metric is called Per Vertex Error (PVE).<\/p>\n\n\n\n<p>(d) Percentage of Correct Parts (PCP): PCP is used to measure the correct detection of limbs. If the distance between the two predicted joint locations and the true limb joint locations is almost less than half of the limb length then the limb is considered detected. However, sometimes it penalizes shorter limbs, for example, a lower arm.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">References<\/h2>\n\n\n\n<p>[1] Lin, Tsung-Yi, et al. \u201cMicrosoft coco: Common objects in context.\u201d&nbsp;<em>ECCV<\/em>&nbsp;2014.<\/p>\n\n\n\n<p>[2] Andriluka et al. \u201c2D Human Pose Estimation: New Benchmark and State of the Art Analysis\u201d CVPR 2014<\/p>\n\n\n\n<p>[3] Ionescu, Catalin, et al. \u201cHuman3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.\u201d&nbsp;<em>PAMI&nbsp;<\/em>2013.<\/p>\n\n\n\n<p>[4] von Marcard, Timo, et al. \u201cRecovering accurate 3d human pose in the wild using imus and a moving camera.\u201d&nbsp;<em>ECCV<\/em>&nbsp;2018.<\/p>\n\n\n\n<p>[5] Patel, Priyanka, et al. \u201cAGORA: Avatars in geography optimized for regression analysis.\u201d&nbsp;<em>CVPR<\/em>&nbsp;2021.<\/p>\n\n\n\n<p>[6] Sun, Yu, et al. \u201cPutting People in their Place: Monocular Regression of 3D People in Depth.\u201d&nbsp;<em>CVPR 2022<\/em>.<\/p>\n\n\n\n<p>[7] Fabbri et al. \u201cLearning to Detect and Track Visible and Occluded Body Joints in a Virtual World\u201d ECCV 2018<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Datasets Some of the datasets we experimented with are as follows: (I) 2D Keypoints datasets COCO keypoint detection[1] MPII Human Pose dataset [2] (II) 3D human mesh datasets Human3.6M [3] 3D Poses in the Wild [4] AGORA [5] Relative Human [6] However, since these datasets aren\u2019t relevant to our target domain, we worked on creating &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Spring 2022&#8221;<\/span><\/a><\/p>\n","protected":false},"author":137,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-311","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Spring 2022 - Modeling and Understanding Pedestrian Behavior<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Spring 2022 - Modeling and Understanding Pedestrian Behavior\" \/>\n<meta property=\"og:description\" content=\"Datasets Some of the datasets we experimented with are as follows: (I) 2D Keypoints datasets COCO keypoint detection[1] MPII Human Pose dataset [2] (II) 3D human mesh datasets Human3.6M [3] 3D Poses in the Wild [4] AGORA [5] Relative Human [6] However, since these datasets aren\u2019t relevant to our target domain, we worked on creating &hellip; Continue reading &quot;Spring 2022&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/\" \/>\n<meta property=\"og:site_name\" content=\"Modeling and Understanding Pedestrian Behavior\" \/>\n<meta property=\"article:modified_time\" content=\"2022-12-21T00:46:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/image-11-1024x341.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/spring-2022\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/spring-2022\\\/\",\"name\":\"Spring 2022 - Modeling and Understanding Pedestrian Behavior\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/spring-2022\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/spring-2022\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/wp-content\\\/uploads\\\/sites\\\/66\\\/2022\\\/12\\\/image-11-1024x341.png\",\"datePublished\":\"2022-12-21T00:44:24+00:00\",\"dateModified\":\"2022-12-21T00:46:29+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/spring-2022\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/spring-2022\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/spring-2022\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/wp-content\\\/uploads\\\/sites\\\/66\\\/2022\\\/12\\\/image-11.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/wp-content\\\/uploads\\\/sites\\\/66\\\/2022\\\/12\\\/image-11.png\",\"width\":1068,\"height\":356},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/spring-2022\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Spring 2022\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/\",\"name\":\"Modeling and Understanding Pedestrian Behavior\",\"description\":\"Students: Adithya Sampath, Mu Chien Hsu | Advisors: Laszlo Jeni (CMU) and Koichiro Niinuma (Fujitsu) | Industry Sponsor: Fujitsu |\",\"publisher\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/#organization\",\"name\":\"Modeling and Understanding Pedestrian Behavior\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/wp-content\\\/uploads\\\/sites\\\/66\\\/2022\\\/05\\\/cropped-download.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/wp-content\\\/uploads\\\/sites\\\/66\\\/2022\\\/05\\\/cropped-download.png\",\"width\":532,\"height\":250,\"caption\":\"Modeling and Understanding Pedestrian Behavior\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team11\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Spring 2022 - Modeling and Understanding Pedestrian Behavior","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/","og_locale":"en_US","og_type":"article","og_title":"Spring 2022 - Modeling and Understanding Pedestrian Behavior","og_description":"Datasets Some of the datasets we experimented with are as follows: (I) 2D Keypoints datasets COCO keypoint detection[1] MPII Human Pose dataset [2] (II) 3D human mesh datasets Human3.6M [3] 3D Poses in the Wild [4] AGORA [5] Relative Human [6] However, since these datasets aren\u2019t relevant to our target domain, we worked on creating &hellip; Continue reading \"Spring 2022\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/","og_site_name":"Modeling and Understanding Pedestrian Behavior","article_modified_time":"2022-12-21T00:46:29+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/image-11-1024x341.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/","name":"Spring 2022 - Modeling and Understanding Pedestrian Behavior","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/image-11-1024x341.png","datePublished":"2022-12-21T00:44:24+00:00","dateModified":"2022-12-21T00:46:29+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/image-11.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/12\/image-11.png","width":1068,"height":356},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/spring-2022\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/"},{"@type":"ListItem","position":2,"name":"Spring 2022"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/","name":"Modeling and Understanding Pedestrian Behavior","description":"Students: Adithya Sampath, Mu Chien Hsu | Advisors: Laszlo Jeni (CMU) and Koichiro Niinuma (Fujitsu) | Industry Sponsor: Fujitsu |","publisher":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/#organization","name":"Modeling and Understanding Pedestrian Behavior","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/#\/schema\/logo\/image\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/05\/cropped-download.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-content\/uploads\/sites\/66\/2022\/05\/cropped-download.png","width":532,"height":250,"caption":"Modeling and Understanding Pedestrian Behavior"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-json\/wp\/v2\/pages\/311","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-json\/wp\/v2\/users\/137"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-json\/wp\/v2\/comments?post=311"}],"version-history":[{"count":2,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-json\/wp\/v2\/pages\/311\/revisions"}],"predecessor-version":[{"id":317,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-json\/wp\/v2\/pages\/311\/revisions\/317"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/wp-json\/wp\/v2\/media?parent=311"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}