{"id":22,"date":"2024-05-08T07:22:35","date_gmt":"2024-05-08T07:22:35","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/?page_id=22"},"modified":"2024-12-09T23:33:24","modified_gmt":"2024-12-09T23:33:24","slug":"project-summary","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/","title":{"rendered":"Project Summary"},"content":{"rendered":"\n

Abstract<\/p>\n\n\n\n

Object permanence in humans is a fundamental cue that helps in understanding persistence of objects, even when they are fully occluded in the scene. Present day methods in object segmentation do not account for this amodal<\/em> nature of the world, and only work for segmentation of visible or modal<\/em> objects. Few amodal methods exist; single-image segmentation methods cannot handle high-levels of occlusions which are better inferred using temporal information, and multi-frame methods have focused solely on segmenting rigid objects. To this end, we propose to tackle video amodal segmentation by formulating it as a conditional generation task, capitalizing on the foundational knowledge in video generative models. Our method is simple; we repurpose these models to condition on a sequence of modal mask frames of an object along with contextual pseudo-depth maps, to learn which object boundary may be occluded and therefore, extended to hallucinate the complete extent of an object. This is followed by a content completion stage which is able to inpaint the occluded regions of an object. Perhaps surprisingly, we do not require RGB (appearance and texture) information, which allows us to better transfer to real-world videos with learning from synthetic datasets. We benchmark our approach alongside a wide array of state-of-the-art methods on four datasets and show a dramatic improvement of upto 13% for amodal segmentation in an object’s occluded region.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Figure 1: An example of video amodal segmentation and content completion<\/p>\n\n\n\n

In this work, we develop a two-stage method that generates object’s amodal (visible + invisible) masks and RGB content. Our method is effectively able to handle severe occlusions and generalizes across diverse object categories, achieving state-of-the-art results on synthetic and real-world datasets. We show one such example of an unseen deformable object category ‘laptop’ that undergoes a complete occlusion in the highlighted<\/mark><\/strong> <\/mark>frame.

<\/p>\n\n\n\n

How to achieve video amodal segmentation and content completion?<\/p>\n\n\n\n

Please refer to our Technical Report<\/a> section.<\/p>\n\n\n\n

For more details:
arXiv: https:\/\/arxiv.org\/abs\/2412.04623<\/a>
project webpage: https:\/\/diffusion-vas.github.io\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Abstract Object permanence in humans is a fundamental cue that helps in understanding persistence of objects, even when they are fully occluded in the scene. Present day methods in object segmentation do not account for this amodal nature of the world, and only work for segmentation of visible or modal objects. Few amodal methods exist; … <\/p>\n

Continue reading “Project Summary”<\/span><\/a><\/p>\n","protected":false},"author":204,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-22","page","type-page","status-publish","hentry"],"yoast_head":"\nProject Summary - Learning diffusion priors for video amodal segmentation<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Project Summary - Learning diffusion priors for video amodal segmentation\" \/>\n<meta property=\"og:description\" content=\"Abstract Object permanence in humans is a fundamental cue that helps in understanding persistence of objects, even when they are fully occluded in the scene. Present day methods in object segmentation do not account for this amodal nature of the world, and only work for segmentation of visible or modal objects. Few amodal methods exist; … Continue reading "Project Summary"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/\" \/>\n<meta property=\"og:site_name\" content=\"Learning diffusion priors for video amodal segmentation\" \/>\n<meta property=\"article:modified_time\" content=\"2024-12-09T23:33:24+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/Figure-teaser-1024x417.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/\",\"name\":\"Project Summary - Learning diffusion priors for video amodal segmentation\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/wp-content\\\/uploads\\\/sites\\\/105\\\/2024\\\/12\\\/Figure-teaser-1024x417.png\",\"datePublished\":\"2024-05-08T07:22:35+00:00\",\"dateModified\":\"2024-12-09T23:33:24+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/wp-content\\\/uploads\\\/sites\\\/105\\\/2024\\\/12\\\/Figure-teaser.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/wp-content\\\/uploads\\\/sites\\\/105\\\/2024\\\/12\\\/Figure-teaser.png\",\"width\":1727,\"height\":704},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Project Summary\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/\",\"name\":\"Learning diffusion priors for video amodal segmentation\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team7\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Project Summary - Learning diffusion priors for video amodal segmentation","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/","og_locale":"en_US","og_type":"article","og_title":"Project Summary - Learning diffusion priors for video amodal segmentation","og_description":"Abstract Object permanence in humans is a fundamental cue that helps in understanding persistence of objects, even when they are fully occluded in the scene. Present day methods in object segmentation do not account for this amodal nature of the world, and only work for segmentation of visible or modal objects. Few amodal methods exist; … Continue reading \"Project Summary\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/","og_site_name":"Learning diffusion priors for video amodal segmentation","article_modified_time":"2024-12-09T23:33:24+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/Figure-teaser-1024x417.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/","name":"Project Summary - Learning diffusion priors for video amodal segmentation","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/Figure-teaser-1024x417.png","datePublished":"2024-05-08T07:22:35+00:00","dateModified":"2024-12-09T23:33:24+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/Figure-teaser.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/Figure-teaser.png","width":1727,"height":704},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/"},{"@type":"ListItem","position":2,"name":"Project Summary"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/","name":"Learning diffusion priors for video amodal segmentation","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-json\/wp\/v2\/pages\/22","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-json\/wp\/v2\/users\/204"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-json\/wp\/v2\/comments?post=22"}],"version-history":[{"count":16,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-json\/wp\/v2\/pages\/22\/revisions"}],"predecessor-version":[{"id":169,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-json\/wp\/v2\/pages\/22\/revisions\/169"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-json\/wp\/v2\/media?parent=22"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}