{"id":8,"date":"2024-05-12T22:51:56","date_gmt":"2024-05-12T22:51:56","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/?page_id=8"},"modified":"2024-12-15T11:18:11","modified_gmt":"2024-12-15T11:18:11","slug":"overview","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/","title":{"rendered":"Overview"},"content":{"rendered":"\n<h3 class=\"wp-block-heading\">Motivation<\/h3>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-28f84493 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:100%\">\n<p class=\"has-text-align-left\">Infering 3D pose and 3D structures of non-rigid deformable objects is of utmost importance in computer vision research. Extracting 3D structure from 2D images or videos is a hard and ambiguous problem, especially for deformable objects like animals and humans. Traditionally 3D pose is obtained using multi-camera rigs or domes which has the ability to capture multi-view images of the objects inside it and infer the 3D structure from it. However, this setup requires accurate camera calibration, is expensive and is not feasible for categories other than humans. With the availabilty of data and deep learning techniques, we intend to build an end-to-end pipeline to obtain the 3D structures with just image or video given as input across multiple categories of rigid and non-rigid objects.<\/p>\n<\/div>\n<\/div>\n\n\n\n<figure class=\"wp-block-image size-large is-style-default\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"437\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-9.01.00\u202fPM-1024x437.png\" alt=\"\" class=\"wp-image-34\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-9.01.00\u202fPM-1024x437.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-9.01.00\u202fPM-300x128.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-9.01.00\u202fPM-768x328.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-9.01.00\u202fPM.png 1175w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-wide\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Applications<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>AR\/VR<\/li>\n\n\n\n<li>Animal\/Human Behaviour Modelling<\/li>\n\n\n\n<li>3D Activity Recognition<\/li>\n\n\n\n<li>Dense 3D Object Reconstruction<\/li>\n\n\n\n<li>Dynamic Deformable Object Splatting<\/li>\n<\/ul>\n\n\n\n<p><\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-wide\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Towards a Universal 3D Lifting Pipeline<\/h3>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"366\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-10.05.50\u202fPM-1024x366.png\" alt=\"\" class=\"wp-image-64\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-10.05.50\u202fPM-1024x366.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-10.05.50\u202fPM-300x107.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-10.05.50\u202fPM-768x275.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-10.05.50\u202fPM-1536x550.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-10.05.50\u202fPM.png 1794w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>The goal is to build a pipeline which can take image or a video of most rigid (boat, airplane, bottle, vase etc.) and non-rigid objects (humans, animals, face, hand etc.) and extract the 3D structure from it. It should have 3 key properties.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>E2E nature &#8211; minimal human supervision<\/li>\n\n\n\n<li>OOD generalization &#8211; foundational model<\/li>\n\n\n\n<li>Temporal consistency &#8211; preserve semantics across frames<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Additionally, can we also reconstruct a mesh?<\/h3>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"306\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img1-1024x306.png\" alt=\"\" class=\"wp-image-138\" style=\"width:518px;height:auto\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img1-1024x306.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img1-300x90.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img1-768x230.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img1.png 1150w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"918\" height=\"736\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img2.png\" alt=\"\" class=\"wp-image-139\" style=\"width:518px;height:auto\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img2.png 918w, https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img2-300x241.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img2-768x616.png 768w\" sizes=\"auto, (max-width: 918px) 100vw, 918px\" \/><\/figure>\n\n\n\n<p>As shown above, the universal image-to-mesh foundational pipeline that we intend to build has three stages. Firstly, 2D keypoints would be extracted from the image input, then the 2D keypoints would be lifted into 3D and lastly a dense 3D mesh would be reconstructed using the 3D keypoints.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Motivation Infering 3D pose and 3D structures of non-rigid deformable objects is of utmost importance in computer vision research. Extracting 3D structure from 2D images or videos is a hard and ambiguous problem, especially for deformable objects like animals and humans. Traditionally 3D pose is obtained using multi-camera rigs or domes which has the ability [&hellip;]<\/p>\n","protected":false},"author":216,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-8","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Overview - Towards Universal 3D Lifting<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Overview - Towards Universal 3D Lifting\" \/>\n<meta property=\"og:description\" content=\"Motivation Infering 3D pose and 3D structures of non-rigid deformable objects is of utmost importance in computer vision research. Extracting 3D structure from 2D images or videos is a hard and ambiguous problem, especially for deformable objects like animals and humans. Traditionally 3D pose is obtained using multi-camera rigs or domes which has the ability [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/\" \/>\n<meta property=\"og:site_name\" content=\"Towards Universal 3D Lifting\" \/>\n<meta property=\"article:modified_time\" content=\"2024-12-15T11:18:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img1-1024x306.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/overview\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/overview\\\/\",\"name\":\"Overview - Towards Universal 3D Lifting\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/overview\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/overview\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/wp-content\\\/uploads\\\/sites\\\/111\\\/2024\\\/12\\\/img1-1024x306.png\",\"datePublished\":\"2024-05-12T22:51:56+00:00\",\"dateModified\":\"2024-12-15T11:18:11+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/overview\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/overview\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/overview\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/wp-content\\\/uploads\\\/sites\\\/111\\\/2024\\\/12\\\/img1.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/wp-content\\\/uploads\\\/sites\\\/111\\\/2024\\\/12\\\/img1.png\",\"width\":1150,\"height\":344},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/overview\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Overview\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/\",\"name\":\"Towards Universal 3D Lifting\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/#organization\",\"name\":\"Towards Universal 3D Lifting\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/wp-content\\\/uploads\\\/sites\\\/111\\\/2024\\\/05\\\/Screenshot-2024-05-12-at-8.26.44\u202fPM.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/wp-content\\\/uploads\\\/sites\\\/111\\\/2024\\\/05\\\/Screenshot-2024-05-12-at-8.26.44\u202fPM.png\",\"width\":588,\"height\":325,\"caption\":\"Towards Universal 3D Lifting\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team13\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Overview - Towards Universal 3D Lifting","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/","og_locale":"en_US","og_type":"article","og_title":"Overview - Towards Universal 3D Lifting","og_description":"Motivation Infering 3D pose and 3D structures of non-rigid deformable objects is of utmost importance in computer vision research. Extracting 3D structure from 2D images or videos is a hard and ambiguous problem, especially for deformable objects like animals and humans. Traditionally 3D pose is obtained using multi-camera rigs or domes which has the ability [&hellip;]","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/","og_site_name":"Towards Universal 3D Lifting","article_modified_time":"2024-12-15T11:18:11+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img1-1024x306.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/","name":"Overview - Towards Universal 3D Lifting","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img1-1024x306.png","datePublished":"2024-05-12T22:51:56+00:00","dateModified":"2024-12-15T11:18:11+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img1.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/12\/img1.png","width":1150,"height":344},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/overview\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/"},{"@type":"ListItem","position":2,"name":"Overview"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/","name":"Towards Universal 3D Lifting","description":"","publisher":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/#organization","name":"Towards Universal 3D Lifting","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/#\/schema\/logo\/image\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-8.26.44\u202fPM.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-content\/uploads\/sites\/111\/2024\/05\/Screenshot-2024-05-12-at-8.26.44\u202fPM.png","width":588,"height":325,"caption":"Towards Universal 3D Lifting"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-json\/wp\/v2\/pages\/8","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-json\/wp\/v2\/users\/216"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-json\/wp\/v2\/comments?post=8"}],"version-history":[{"count":19,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-json\/wp\/v2\/pages\/8\/revisions"}],"predecessor-version":[{"id":140,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-json\/wp\/v2\/pages\/8\/revisions\/140"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team13\/wp-json\/wp\/v2\/media?parent=8"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}