{"id":33,"date":"2024-05-12T16:55:07","date_gmt":"2024-05-12T16:55:07","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/?page_id=33"},"modified":"2024-12-08T19:56:44","modified_gmt":"2024-12-08T19:56:44","slug":"experiments","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/","title":{"rendered":"Experiments"},"content":{"rendered":"\n

Evaluating the correlation of automated metrics with
human ratings on GenAI-Bench<\/strong><\/h1>\n\n\n\n

We report higher scores in Pairwise accuracy, Pearson, and Kendall indicating better performance. VQAScore, using the CLIP-FlanT5 VQA model, achieves the strongest agreement with human ratings on images and videos, significantly surpassing metrics like CLIPScore, PickScore, and Davidsonian.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Boosting Text-to-Visual Generation with VQAScore: A Comparative Analysis<\/strong><\/h1>\n\n\n\n

We enhanced text-to-visual generation by evaluating nine candidate images, marking performance improvements in green and declines in red. Selecting images with the highest VQAS score significantly increases human alignment ratings. Conversely, ranking by CLIPScore may yield the same or reduced performance. VQAScore is 2x to 3x more effective than methods like PickScore, which require expensive human feedback, or those that decompose texts using ChatGPT (Davidsonian). Table 3 shows performance improvements for various scoring methods across basic, advanced, and all prompts.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Enhancing DALL-E 3 Image Generation with VQAScore<\/strong><\/h1>\n\n\n\n

Ranking DALL-E 3 generated images with VQAScore and CLIPScore reveals that VQAScore surpasses CLIPScore, especially for prompts involving attributes, relationships, and higher-order reasoning. This demonstrates VQAScore\u2019s potential to enhance text-to-image generation solely with an image generation API. We provide detailed performance gains for VQAScore and other metrics.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Improve DALLE3 and SDXL by image ranking<\/strong><\/h1>\n\n\n\n

We present the average human ratings of 7 popular scoring methods across basic, advanced, and all prompts on GenAI-Bench. Performance gains over the Random baseline (no ranking) are highlighted in green, while decreases are marked in red.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Alignment on Text-to-Video Models with Cinematic T2V Benchmark<\/strong><\/h1>\n\n\n\n

We present an evaluation using our cinematic T2V benchmark, which incorporates camera comments that are often overlooked in current video captions. We help T2V models generate visually consistent and realistic videos by bringing rich captions.<\/p>\n\n\n\n

\"\"<\/figure>\n","protected":false},"excerpt":{"rendered":"

Evaluating the correlation of automated metrics withhuman ratings on GenAI-Bench We report higher scores in Pairwise accuracy, Pearson, and Kendall indicating better performance. VQAScore, using the CLIP-FlanT5 VQA model, achieves the strongest agreement with human ratings on images and videos, significantly surpassing metrics like CLIPScore, PickScore, and Davidsonian. Boosting Text-to-Visual Generation with VQAScore: A Comparative … <\/p>\n

Continue reading “Experiments”<\/span><\/a><\/p>\n","protected":false},"author":205,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-33","page","type-page","status-publish","hentry"],"yoast_head":"\nExperiments - Alignment for Vision-Language Foundation Models<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Experiments - Alignment for Vision-Language Foundation Models\" \/>\n<meta property=\"og:description\" content=\"Evaluating the correlation of automated metrics withhuman ratings on GenAI-Bench We report higher scores in Pairwise accuracy, Pearson, and Kendall indicating better performance. VQAScore, using the CLIP-FlanT5 VQA model, achieves the strongest agreement with human ratings on images and videos, significantly surpassing metrics like CLIPScore, PickScore, and Davidsonian. Boosting Text-to-Visual Generation with VQAScore: A Comparative … Continue reading "Experiments"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/\" \/>\n<meta property=\"og:site_name\" content=\"Alignment for Vision-Language Foundation Models\" \/>\n<meta property=\"article:modified_time\" content=\"2024-12-08T19:56:44+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715384326555.png?w=569\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/experiments\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/experiments\\\/\",\"name\":\"Experiments - Alignment for Vision-Language Foundation Models\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/experiments\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/experiments\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojectsricmu.files.wordpress.com\\\/2024\\\/05\\\/1715384326555.png?w=569\",\"datePublished\":\"2024-05-12T16:55:07+00:00\",\"dateModified\":\"2024-12-08T19:56:44+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/experiments\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/experiments\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/experiments\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojectsricmu.files.wordpress.com\\\/2024\\\/05\\\/1715384326555.png?w=569\",\"contentUrl\":\"https:\\\/\\\/mscvprojectsricmu.files.wordpress.com\\\/2024\\\/05\\\/1715384326555.png?w=569\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/experiments\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Experiments\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/\",\"name\":\"Alignment for Vision-Language Foundation Models\",\"description\":\"Students: Yixin Fei, Kewen Wu, Pengliang Ji | Advisors: Zhiqiu Lin, Deva Ramanan (Carnegie Mellon University, Robotics Institute)\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2024team8\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Experiments - Alignment for Vision-Language Foundation Models","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/","og_locale":"en_US","og_type":"article","og_title":"Experiments - Alignment for Vision-Language Foundation Models","og_description":"Evaluating the correlation of automated metrics withhuman ratings on GenAI-Bench We report higher scores in Pairwise accuracy, Pearson, and Kendall indicating better performance. VQAScore, using the CLIP-FlanT5 VQA model, achieves the strongest agreement with human ratings on images and videos, significantly surpassing metrics like CLIPScore, PickScore, and Davidsonian. Boosting Text-to-Visual Generation with VQAScore: A Comparative … Continue reading \"Experiments\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/","og_site_name":"Alignment for Vision-Language Foundation Models","article_modified_time":"2024-12-08T19:56:44+00:00","og_image":[{"url":"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715384326555.png?w=569","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/","name":"Experiments - Alignment for Vision-Language Foundation Models","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715384326555.png?w=569","datePublished":"2024-05-12T16:55:07+00:00","dateModified":"2024-12-08T19:56:44+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/#primaryimage","url":"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715384326555.png?w=569","contentUrl":"https:\/\/mscvprojectsricmu.files.wordpress.com\/2024\/05\/1715384326555.png?w=569"},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/experiments\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/"},{"@type":"ListItem","position":2,"name":"Experiments"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/","name":"Alignment for Vision-Language Foundation Models","description":"Students: Yixin Fei, Kewen Wu, Pengliang Ji | Advisors: Zhiqiu Lin, Deva Ramanan (Carnegie Mellon University, Robotics Institute)","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/pages\/33","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/users\/205"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/comments?post=33"}],"version-history":[{"count":3,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/pages\/33\/revisions"}],"predecessor-version":[{"id":95,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/pages\/33\/revisions\/95"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2024team8\/wp-json\/wp\/v2\/media?parent=33"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}