{"id":29,"date":"2024-05-12T16:53:19","date_gmt":"2024-05-12T16:53:19","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/?page_id=29"},"modified":"2024-12-08T19:34:35","modified_gmt":"2024-12-08T19:34:35","slug":"about","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/","title":{"rendered":"About"},"content":{"rendered":"\n<p>Text-to-visual models, which now create realistic images and videos, face challenges with complex prompts involving attributes, relationships, and advanced reasoning. Our study on GenAI-Bench evaluates top image and video generators for these tasks. We compare automated evaluation metrics against our collected human ratings and find that VQAScore significantly outperforms previous metrics such as CLIPScore. In addition, VQAScore can improve generation in a black-box manner (without fine-tuning) by simply ranking a few (3 to 9) candidate images. It is 2-3 times more effective than methods such as PickScore and ImageReward in improving human ratings for models like DALL-E 3 and Stable Diffusion.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715380758753.png?w=1024\" alt=\"\" class=\"wp-image-162\" \/><\/figure>\n\n\n\n<h1 class=\"wp-block-heading\"><strong>Motivation<\/strong><\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Text-to-visual models struggle with generating images involving compositional text prompts.&nbsp;<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715380703443.png?w=1024\" alt=\"\" class=\"wp-image-161\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Evaluating alignment robustly is challenging for both automatic metrics and human evaluation.<\/li>\n\n\n\n<li>The automated evaluation metrics for alignment often function as a bag of words.<\/li>\n\n\n\n<li>Text-to-video models struggle to understand camera aspects and 3D motions.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715381926374.png?w=1024\" alt=\"\" class=\"wp-image-179\" \/><\/figure>\n\n\n\n<h1 class=\"wp-block-heading\"><strong>Goal<\/strong><\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Providing<\/strong> reliable scores, VQAScore, to evaluate complex prompt alignment without relying on expensive human feedback.<\/li>\n\n\n\n<li><strong>Building <\/strong>a benchmark, GenAI-Bench, to assess essential visio-linguistic compositional reasoning skills for text-to-visual generative models and vision-language alignment metrics.<\/li>\n\n\n\n<li><strong>Creating <\/strong>a high-quality text-video paired dataset, incorporating aspects like shot composition, camera movements, and lighting effects.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-us.googleusercontent.com\/1f_mfT3rOroydIGLIvnKWNB442ecyae89SJhaGbuPKC3yqljFUyemwVUHSgrHO-1VkIIMhi6qeWkqwt_KJkDP6N8UzLH3yL6Yj0UtG0Wt0I36yWS3zZkDaGUSJ3kj5Y78OBpSPXhf4wPsn0hu59fkXizTA=s2048\" alt=\"\" \/><\/figure>\n\n\n\n<h1 class=\"wp-block-heading\"><strong>Overview<\/strong><\/h1>\n\n\n\n<p>In words, our work explores:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>A Reliable Alignment Metric for Text-to-Visual Models<\/strong>\n<ul class=\"wp-block-list\">\n<li>We Introduced a yes\/no question-based metric with FlanT5 for robust text-to-visual alignment evaluation.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>A Visio-linguistic Compositional<\/strong> <strong>Benchmark for Text-to-Image Models<\/strong>\n<ul class=\"wp-block-list\">\n<li>We constructed 1,600 compositional prompts and VQAScore to rank outputs, boosting alignment validated by 43,200 human ratings.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>A Cinematic Benchmark for Text-to-Video Models<\/strong>\n<ul class=\"wp-block-list\">\n<li>We established benchmarks to evaluate visio-linguistic reasoning, integrating shot composition, movements, and lighting for realistic outputs.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>Text-to-visual models, which now create realistic images and videos, face challenges with complex prompts involving attributes, relationships, and advanced reasoning. Our study on GenAI-Bench evaluates top image and video generators for these tasks. We compare automated evaluation metrics against our collected human ratings and find that VQAScore significantly outperforms previous metrics such as CLIPScore. In &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;About&#8221;<\/span><\/a><\/p>\n","protected":false},"author":206,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-29","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>About - Alignment for Vision-Language Foundation Models<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"About - Alignment for Vision-Language Foundation Models\" \/>\n<meta property=\"og:description\" content=\"Text-to-visual models, which now create realistic images and videos, face challenges with complex prompts involving attributes, relationships, and advanced reasoning. Our study on GenAI-Bench evaluates top image and video generators for these tasks. We compare automated evaluation metrics against our collected human ratings and find that VQAScore significantly outperforms previous metrics such as CLIPScore. In &hellip; Continue reading &quot;About&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/\" \/>\n<meta property=\"og:site_name\" content=\"Alignment for Vision-Language Foundation Models\" \/>\n<meta property=\"article:modified_time\" content=\"2024-12-08T19:34:35+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715380758753.png?w=1024\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/\",\"name\":\"About - Alignment for Vision-Language Foundation Models\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojectsricmu.files.wordpress.com\\\/2024\\\/05\\\/1715380758753.png?w=1024\",\"datePublished\":\"2024-05-12T16:53:19+00:00\",\"dateModified\":\"2024-12-08T19:34:35+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojectsricmu.files.wordpress.com\\\/2024\\\/05\\\/1715380758753.png?w=1024\",\"contentUrl\":\"https:\\\/\\\/mscvprojectsricmu.files.wordpress.com\\\/2024\\\/05\\\/1715380758753.png?w=1024\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"About\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/\",\"name\":\"Alignment for Vision-Language Foundation Models\",\"description\":\"Students: Yixin Fei, Kewen Wu, Pengliang Ji | Advisors: Zhiqiu Lin, Deva Ramanan (Carnegie Mellon University, Robotics Institute)\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"About - Alignment for Vision-Language Foundation Models","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/","og_locale":"en_US","og_type":"article","og_title":"About - Alignment for Vision-Language Foundation Models","og_description":"Text-to-visual models, which now create realistic images and videos, face challenges with complex prompts involving attributes, relationships, and advanced reasoning. Our study on GenAI-Bench evaluates top image and video generators for these tasks. We compare automated evaluation metrics against our collected human ratings and find that VQAScore significantly outperforms previous metrics such as CLIPScore. In &hellip; Continue reading \"About\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/","og_site_name":"Alignment for Vision-Language Foundation Models","article_modified_time":"2024-12-08T19:34:35+00:00","og_image":[{"url":"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715380758753.png?w=1024","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/","name":"About - Alignment for Vision-Language Foundation Models","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715380758753.png?w=1024","datePublished":"2024-05-12T16:53:19+00:00","dateModified":"2024-12-08T19:34:35+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/#primaryimage","url":"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715380758753.png?w=1024","contentUrl":"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715380758753.png?w=1024"},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/"},{"@type":"ListItem","position":2,"name":"About"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/","name":"Alignment for Vision-Language Foundation Models","description":"Students: Yixin Fei, Kewen Wu, Pengliang Ji | Advisors: Zhiqiu Lin, Deva Ramanan (Carnegie Mellon University, Robotics Institute)","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/pages\/29","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/users\/206"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/comments?post=29"}],"version-history":[{"count":5,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/pages\/29\/revisions"}],"predecessor-version":[{"id":84,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/pages\/29\/revisions\/84"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/media?parent=29"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}