{"id":25,"date":"2023-05-09T03:17:40","date_gmt":"2023-05-09T03:17:40","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/?page_id=25"},"modified":"2023-12-18T19:03:29","modified_gmt":"2023-12-18T19:03:29","slug":"introduction","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/","title":{"rendered":"Introduction"},"content":{"rendered":"\n<h1 class=\"wp-block-heading has-text-align-left\"><mark class=\"has-inline-color has-vivid-cyan-blue-color\">Motivation<\/mark><\/h1>\n\n\n\n<p>There has been a recent boom in the area of text-to-image generation, with <mark class=\"has-inline-color has-vivid-red-color\"><a href=\"https:\/\/ommer-lab.com\/research\/latent-diffusion-models\/\">Stable Diffusion<\/a><\/mark>, <mark class=\"has-inline-color has-vivid-red-color\"><a href=\"https:\/\/openai.com\/product\/dall-e-2\">DALL\u00b7E 2<\/a><\/mark>, and <mark class=\"has-inline-color has-vivid-red-color\"><a href=\"https:\/\/arxiv.org\/abs\/2205.11487\">Imagen<\/a><\/mark> allowing users to generate photo-realistic and highly detailed images from just text queries. There also has been a lot of recent work on the more challenging area of text-to-video synthesis, namely, <mark class=\"has-inline-color has-vivid-red-color\"><a href=\"https:\/\/imagen.research.google\/video\/\">Imagen Video<\/a><\/mark> and <mark class=\"has-inline-color has-vivid-red-color\"><a href=\"https:\/\/makeavideo.studio\/\">Make-A-Video<\/a><\/mark>. However, the area of text-to-cinemagraph, or infinitely looping aesthetic videos (also termed cinemagraphs) is relatively unexplored.<\/p>\n\n\n\n<p>If we were to enhance the ability of these text-to-image models that can generate high-fidelity images coherent with text prompts to generate high-quality animations directly from a text query, it would enable professionals a very easy medium to create video assets for movies, advertisements, games, etc., rather than having to spend enormous amounts of time doing it manually. In this work, we focus mainly on creating infinitely seamlessly looping videos for scenes like a waterfall, rivers, and oceans, which can act as background assets. <\/p>\n\n\n\n<p>Additionally, with the exponential rise in popularity of short video social media platforms like Instagram, TikTok, etc, text-to-cinemagraph also seems to be a very lucrative tool for amateurs.<\/p>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"992\" style=\"aspect-ratio: 2462 \/ 992;\" width=\"2462\" autoplay controls loop src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/wp-content\/uploads\/sites\/90\/2023\/12\/teaser.mp4\"><\/video><figcaption class=\"wp-element-caption\"><strong>Figure 1:\u00a0<\/strong>Cinemagraph (looping video) generated from text caption<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>Motivation There has been a recent boom in the area of text-to-image generation, with Stable Diffusion, DALL\u00b7E 2, and Imagen allowing users to generate photo-realistic and highly detailed images from just text queries. There also has been a lot of recent work on the more challenging area of text-to-video synthesis, namely, Imagen Video and Make-A-Video. &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Introduction&#8221;<\/span><\/a><\/p>\n","protected":false},"author":177,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-25","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Introduction - Adapting Stable Diffusion Models for Video Synthesis<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Introduction - Adapting Stable Diffusion Models for Video Synthesis\" \/>\n<meta property=\"og:description\" content=\"Motivation There has been a recent boom in the area of text-to-image generation, with Stable Diffusion, DALL\u00b7E 2, and Imagen allowing users to generate photo-realistic and highly detailed images from just text queries. There also has been a lot of recent work on the more challenging area of text-to-video synthesis, namely, Imagen Video and Make-A-Video. &hellip; Continue reading &quot;Introduction&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/\" \/>\n<meta property=\"og:site_name\" content=\"Adapting Stable Diffusion Models for Video Synthesis\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-18T19:03:29+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team13\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team13\\\/\",\"name\":\"Introduction - Adapting Stable Diffusion Models for Video Synthesis\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team13\\\/#website\"},\"datePublished\":\"2023-05-09T03:17:40+00:00\",\"dateModified\":\"2023-12-18T19:03:29+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team13\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team13\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team13\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team13\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Introduction\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team13\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team13\\\/\",\"name\":\"Adapting Stable Diffusion Models for Video Synthesis\",\"description\":\"Students: Aniruddha Mahapatra | Advisors: Jun-Yan Zhu (CMU), Aliaksandr Sairohin (Snap), Hsin-Ying Lee (Snap), Sergey Tulyakov (Snap) | Sponsor: CMU\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team13\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Introduction - Adapting Stable Diffusion Models for Video Synthesis","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/","og_locale":"en_US","og_type":"article","og_title":"Introduction - Adapting Stable Diffusion Models for Video Synthesis","og_description":"Motivation There has been a recent boom in the area of text-to-image generation, with Stable Diffusion, DALL\u00b7E 2, and Imagen allowing users to generate photo-realistic and highly detailed images from just text queries. There also has been a lot of recent work on the more challenging area of text-to-video synthesis, namely, Imagen Video and Make-A-Video. &hellip; Continue reading \"Introduction\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/","og_site_name":"Adapting Stable Diffusion Models for Video Synthesis","article_modified_time":"2023-12-18T19:03:29+00:00","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/","name":"Introduction - Adapting Stable Diffusion Models for Video Synthesis","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/#website"},"datePublished":"2023-05-09T03:17:40+00:00","dateModified":"2023-12-18T19:03:29+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/"},{"@type":"ListItem","position":2,"name":"Introduction"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/","name":"Adapting Stable Diffusion Models for Video Synthesis","description":"Students: Aniruddha Mahapatra | Advisors: Jun-Yan Zhu (CMU), Aliaksandr Sairohin (Snap), Hsin-Ying Lee (Snap), Sergey Tulyakov (Snap) | Sponsor: CMU","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/wp-json\/wp\/v2\/pages\/25","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/wp-json\/wp\/v2\/users\/177"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/wp-json\/wp\/v2\/comments?post=25"}],"version-history":[{"count":12,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/wp-json\/wp\/v2\/pages\/25\/revisions"}],"predecessor-version":[{"id":138,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/wp-json\/wp\/v2\/pages\/25\/revisions\/138"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team13\/wp-json\/wp\/v2\/media?parent=25"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}