{"id":40,"date":"2025-05-06T05:47:29","date_gmt":"2025-05-06T05:47:29","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/?page_id=40"},"modified":"2025-05-08T05:04:15","modified_gmt":"2025-05-08T05:04:15","slug":"benchmark","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/","title":{"rendered":"Benchmark"},"content":{"rendered":"\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"454\" src=\"http:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_overview-1024x454.png\" alt=\"\" class=\"wp-image-65\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_overview-1024x454.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_overview-300x133.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_overview-768x340.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_overview-1536x681.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_overview.png 1606w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><figcaption class=\"wp-element-caption\">Overview of VLM2-Bench<\/figcaption><\/figure>\n\n\n\n<p>Driven by this goal, we build upon\u00a0<strong>VLM\u00b2-Bench<\/strong> (<a href=\"https:\/\/vlm2-bench.github.io\/\">https:\/\/vlm2-bench.gith<\/a>ub.io\/), a benchmark developed in prior work by one of our authors, Dongyu Yao. VLM\u00b2-Bench is specifically designed to evaluate whether Vision-Language Models can\u00a0<strong>visually link matching <\/strong><span style=\"margin: 0px;padding: 0px\"><strong>cues\u00a0<\/strong>across<\/span> multiple images and videos.<\/p>\n\n\n\n<p>To comprehensively assess this ability, the benchmark covers&nbsp;<strong>three major categories of visual cues<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>General Cues (GC)<\/strong><\/li>\n\n\n\n<li><strong>Object-centric Cues (OC)<\/strong><\/li>\n\n\n\n<li><strong>Person-centric Cues (PC)<\/strong><\/li>\n<\/ul>\n\n\n\n<p>It includes\u00a0<strong>9 sub-tasks<\/strong>, featuring both multi-image sequences and video-based scenarios, and comprises a total of\u00a0<strong>3,060 VQA test cases<\/strong>, providing a thorough examination of VLMs\u2019 core visual linking capabilities.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"485\" src=\"http:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_res-1-1024x485.png\" alt=\"\" class=\"wp-image-67\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_res-1-1024x485.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_res-1-300x142.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_res-1-768x364.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_res-1-1536x728.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_res-1.png 1540w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><\/figure>\n\n\n\n<p>To contextualize model performance, this benchmark introduce two baselines:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Chance-Level<\/strong>, representing random guessing<\/li>\n\n\n\n<li><strong>Human-Level<\/strong>, reflecting natural visual linking ability<\/li>\n<\/ul>\n\n\n\n<p>As shown in\u00a0<strong>Table 1<\/strong>, humans find VLM\u00b2-Bench tasks relatively easy. However, most state-of-the-art models not only fall\u00a0<strong>far short of human performance<\/strong>, but in many cases perform\u00a0<strong>worse than random guessing<\/strong>. This performance gap is especially pronounced in the\u00a0<strong>VID<\/strong>\u00a0task, which requires tracking and describing people across video frames. Models frequently\u00a0<strong>mistake different individuals as the same person<\/strong> or fail to recognize reappearing individuals.<\/p>\n\n\n\n<p>Interestingly, models show relatively\u00a0<strong>better performance on Person-centric Cues (PC)<\/strong>\u00a0compared to\u00a0<strong>Object-centric Cues (OC)<\/strong>. We hypothesize this is due to the\u00a0<strong>textual anchors<\/strong>\u00a0available in PC tasks\u2014such as proper names\u2014which offer strong and consistent visual associations. In contrast, OC tasks often involve generic category labels (e.g., &#8220;bag&#8221;, &#8220;bottle&#8221;), which offer weaker anchoring and make fine-grained object linking much harder.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udd0d Key Findings from VLM2-Bench<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Language aids vision\u2014but not always enough<\/strong><br>While language-based reasoning can help models make logical connections, it is&nbsp;<strong>not sufficient<\/strong>&nbsp;for fine-grained visual matching without strong visual grounding.<\/li>\n\n\n\n<li><strong>Visual prompting needs a stronger vision-side ability<\/strong><br>The effectiveness of visual prompts hinges on whether models can truly understand\u00a0<strong>both<\/strong>\u00a0the visual content and the prompt, not just rely on language cues.<\/li>\n\n\n\n<li><strong>Person vs. Object: Not all cues are equal<\/strong><br>Visual prompting performs&nbsp;<strong>better on object-centric cues<\/strong>&nbsp;than on person-centric ones. This suggests that current models may rely more on textual anchors (e.g., names) rather than purely visual identity.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n","protected":false},"excerpt":{"rendered":"<p>Driven by this goal, we build upon\u00a0VLM\u00b2-Bench (https:\/\/vlm2-bench.github.io\/), a benchmark developed in prior work by one of our authors, Dongyu Yao. VLM\u00b2-Bench is specifically designed to evaluate whether Vision-Language Models can\u00a0visually link matching cues\u00a0across multiple images and videos. To comprehensively assess this ability, the benchmark covers&nbsp;three major categories of visual cues: It includes\u00a09 sub-tasks, featuring &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Benchmark&#8221;<\/span><\/a><\/p>\n","protected":false},"author":232,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-40","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Benchmark - LVLM Multi-image VQA Dehallucination<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Benchmark - LVLM Multi-image VQA Dehallucination\" \/>\n<meta property=\"og:description\" content=\"Driven by this goal, we build upon\u00a0VLM\u00b2-Bench (https:\/\/vlm2-bench.github.io\/), a benchmark developed in prior work by one of our authors, Dongyu Yao. VLM\u00b2-Bench is specifically designed to evaluate whether Vision-Language Models can\u00a0visually link matching cues\u00a0across multiple images and videos. To comprehensively assess this ability, the benchmark covers&nbsp;three major categories of visual cues: It includes\u00a09 sub-tasks, featuring &hellip; Continue reading &quot;Benchmark&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/\" \/>\n<meta property=\"og:site_name\" content=\"LVLM Multi-image VQA Dehallucination\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-08T05:04:15+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_overview-1024x454.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/benchmark\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/benchmark\\\/\",\"name\":\"Benchmark - LVLM Multi-image VQA Dehallucination\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/benchmark\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/benchmark\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/wp-content\\\/uploads\\\/sites\\\/119\\\/2025\\\/05\\\/bench_overview-1024x454.png\",\"datePublished\":\"2025-05-06T05:47:29+00:00\",\"dateModified\":\"2025-05-08T05:04:15+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/benchmark\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/benchmark\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/benchmark\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/wp-content\\\/uploads\\\/sites\\\/119\\\/2025\\\/05\\\/bench_overview.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/wp-content\\\/uploads\\\/sites\\\/119\\\/2025\\\/05\\\/bench_overview.png\",\"width\":1606,\"height\":712},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/benchmark\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Benchmark\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/\",\"name\":\"LVLM Multi-image VQA Dehallucination\",\"description\":\"Authors: Dongyu Yao, Keling Yao, Harry Chi; Advisor: Prof. Katia Sycara and Dr. Yaqi Xie\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Benchmark - LVLM Multi-image VQA Dehallucination","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/","og_locale":"en_US","og_type":"article","og_title":"Benchmark - LVLM Multi-image VQA Dehallucination","og_description":"Driven by this goal, we build upon\u00a0VLM\u00b2-Bench (https:\/\/vlm2-bench.github.io\/), a benchmark developed in prior work by one of our authors, Dongyu Yao. VLM\u00b2-Bench is specifically designed to evaluate whether Vision-Language Models can\u00a0visually link matching cues\u00a0across multiple images and videos. To comprehensively assess this ability, the benchmark covers&nbsp;three major categories of visual cues: It includes\u00a09 sub-tasks, featuring &hellip; Continue reading \"Benchmark\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/","og_site_name":"LVLM Multi-image VQA Dehallucination","article_modified_time":"2025-05-08T05:04:15+00:00","og_image":[{"url":"http:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_overview-1024x454.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/","name":"Benchmark - LVLM Multi-image VQA Dehallucination","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/#primaryimage"},"thumbnailUrl":"http:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_overview-1024x454.png","datePublished":"2025-05-06T05:47:29+00:00","dateModified":"2025-05-08T05:04:15+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_overview.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-content\/uploads\/sites\/119\/2025\/05\/bench_overview.png","width":1606,"height":712},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/benchmark\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/"},{"@type":"ListItem","position":2,"name":"Benchmark"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/","name":"LVLM Multi-image VQA Dehallucination","description":"Authors: Dongyu Yao, Keling Yao, Harry Chi; Advisor: Prof. Katia Sycara and Dr. Yaqi Xie","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/pages\/40","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/users\/232"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/comments?post=40"}],"version-history":[{"count":4,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/pages\/40\/revisions"}],"predecessor-version":[{"id":70,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/pages\/40\/revisions\/70"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/media?parent=40"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}