{"id":5,"date":"2025-05-06T03:44:03","date_gmt":"2025-05-06T03:44:03","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/?page_id=5"},"modified":"2025-05-08T05:16:15","modified_gmt":"2025-05-08T05:16:15","slug":"metrics","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/metrics\/","title":{"rendered":"Evaluation Metrics"},"content":{"rendered":"\n

However, the original VLM2<\/sup>-Bench only evaluates LVLMs with high-level questions like multi-choice, yes\/no, and counting. These questions are too general to closely assess whether the model can truly understand the visual cues and link them properly, so we need more fine-grained inspections on these images to evaluate hallucination.<\/p>\n\n\n\n

In the field of hallucination evaluation, there is a very popular paper called POPE, P<\/strong>olling-based O<\/strong>bject P<\/strong>robing E<\/strong>valuation (Li et al., EMNLP 2023<\/a>), which was proposed to evaluate object-level hallucination of single images. It generates ground-truth objects in the image via an off-the-shelf segmentation network and human annotation, and samples non-existent objects via negative sampling from a pre-defined object pool. After getting ground-truth and non-existent objects, a bunch of yes\/no questions are generated based on each object. In this way, we can easily evaluate whether an LVLM truly understands the image content in a relatively easier manner, because we don\u2019t need to come up with complicated semantic parsing algorithms to deal with open-ended responses<\/p>\n\n\n\n

\"\"
Figure 1: The process of constructing the hallucination metric of POPE, targeting the task of single-image object-level hallucination of LVLM.<\/figcaption><\/figure>\n\n\n\n

Inspired by this paper, we propose our metric on the VLM2<\/sup>-Bench dataset. Given an image sequence, we first manually extract the key attributes to distinguish between images, like color, text, and number, and then form yes\/no questions following the template. The ground truth answer is annotated by humans. By doing this, each image sequence now has a fine-grained question set Q<\/p>\n\n\n\n

\"\"
Figure 2: Pipeline for constructing fine-grained yes\/no questions on VLM^2-Bench.<\/figcaption><\/figure>\n\n\n\n

When evaluating LVLM, we need to calculate a hallucination score for each image sequence, and the overall score of the whole task, like counting or multi-choice, is just the summation over all s. To do this, we first ask the coarse-grained question, like “How many distinct shirts are there in these images?”. If the answer is incorrect, the hallucination score of this image sequence is 0 because it cannot even answer the high-level question correctly. Otherwise, we go through each fine-grained question, answer it one by one, and calculate the overall accuracy as the hallucination score.<\/p>\n\n\n\n

\"\"
Figure 3: Pipeline for calculating the hallucination score of each image sequence.<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"

However, the original VLM2-Bench only evaluates LVLMs with high-level questions like multi-choice, yes\/no, and counting. These questions are too general to closely assess whether the model can truly understand the visual cues and link them properly, so we need more fine-grained inspections on these images to evaluate hallucination. In the field of hallucination evaluation, there … <\/p>\n

Continue reading “Evaluation Metrics”<\/span><\/a><\/p>\n","protected":false},"author":230,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":"[]"},"class_list":["post-5","page","type-page","status-publish","hentry"],"yoast_head":"\nEvaluation Metrics - LVLM Multi-image VQA Dehallucination<\/title>\n<meta name=\"robots\" content=\"noindex, follow\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Evaluation Metrics - LVLM Multi-image VQA Dehallucination\" \/>\n<meta property=\"og:description\" content=\"However, the original VLM2-Bench only evaluates LVLMs with high-level questions like multi-choice, yes\/no, and counting. These questions are too general to closely assess whether the model can truly understand the visual cues and link them properly, so we need more fine-grained inspections on these images to evaluate hallucination. In the field of hallucination evaluation, there … Continue reading "Evaluation Metrics"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/metrics\/\" \/>\n<meta property=\"og:site_name\" content=\"LVLM Multi-image VQA Dehallucination\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-08T05:16:15+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/lh7-rt.googleusercontent.com\/slidesz\/AGV_vUdHl7VUQH4WKEAilEDMJt21shw-liTP5Slikbft35EBSyLbvWUVCNHwnCwQbX79jmfOEYu1HfPgBHOnAP3ZtkWJprPqjYSVKWVQS8rmN8U3BfzZRRRwYkOHYBOk2iTvxabhJaIi=s2048?key=MhxsOoUwM9UCIcXKZDmfhW2h\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/metrics\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/metrics\\\/\",\"name\":\"Evaluation Metrics - LVLM Multi-image VQA Dehallucination\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/metrics\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/metrics\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/lh7-rt.googleusercontent.com\\\/slidesz\\\/AGV_vUdHl7VUQH4WKEAilEDMJt21shw-liTP5Slikbft35EBSyLbvWUVCNHwnCwQbX79jmfOEYu1HfPgBHOnAP3ZtkWJprPqjYSVKWVQS8rmN8U3BfzZRRRwYkOHYBOk2iTvxabhJaIi=s2048?key=MhxsOoUwM9UCIcXKZDmfhW2h\",\"datePublished\":\"2025-05-06T03:44:03+00:00\",\"dateModified\":\"2025-05-08T05:16:15+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/metrics\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/metrics\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/metrics\\\/#primaryimage\",\"url\":\"https:\\\/\\\/lh7-rt.googleusercontent.com\\\/slidesz\\\/AGV_vUdHl7VUQH4WKEAilEDMJt21shw-liTP5Slikbft35EBSyLbvWUVCNHwnCwQbX79jmfOEYu1HfPgBHOnAP3ZtkWJprPqjYSVKWVQS8rmN8U3BfzZRRRwYkOHYBOk2iTvxabhJaIi=s2048?key=MhxsOoUwM9UCIcXKZDmfhW2h\",\"contentUrl\":\"https:\\\/\\\/lh7-rt.googleusercontent.com\\\/slidesz\\\/AGV_vUdHl7VUQH4WKEAilEDMJt21shw-liTP5Slikbft35EBSyLbvWUVCNHwnCwQbX79jmfOEYu1HfPgBHOnAP3ZtkWJprPqjYSVKWVQS8rmN8U3BfzZRRRwYkOHYBOk2iTvxabhJaIi=s2048?key=MhxsOoUwM9UCIcXKZDmfhW2h\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/metrics\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Evaluation Metrics\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/\",\"name\":\"LVLM Multi-image VQA Dehallucination\",\"description\":\"Authors: Dongyu Yao, Keling Yao, Harry Chi; Advisor: Prof. Katia Sycara and Dr. Yaqi Xie\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team15\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Evaluation Metrics - LVLM Multi-image VQA Dehallucination","robots":{"index":"noindex","follow":"follow"},"og_locale":"en_US","og_type":"article","og_title":"Evaluation Metrics - LVLM Multi-image VQA Dehallucination","og_description":"However, the original VLM2-Bench only evaluates LVLMs with high-level questions like multi-choice, yes\/no, and counting. These questions are too general to closely assess whether the model can truly understand the visual cues and link them properly, so we need more fine-grained inspections on these images to evaluate hallucination. In the field of hallucination evaluation, there … Continue reading \"Evaluation Metrics\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/metrics\/","og_site_name":"LVLM Multi-image VQA Dehallucination","article_modified_time":"2025-05-08T05:16:15+00:00","og_image":[{"url":"https:\/\/lh7-rt.googleusercontent.com\/slidesz\/AGV_vUdHl7VUQH4WKEAilEDMJt21shw-liTP5Slikbft35EBSyLbvWUVCNHwnCwQbX79jmfOEYu1HfPgBHOnAP3ZtkWJprPqjYSVKWVQS8rmN8U3BfzZRRRwYkOHYBOk2iTvxabhJaIi=s2048?key=MhxsOoUwM9UCIcXKZDmfhW2h","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/metrics\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/metrics\/","name":"Evaluation Metrics - LVLM Multi-image VQA Dehallucination","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/metrics\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/metrics\/#primaryimage"},"thumbnailUrl":"https:\/\/lh7-rt.googleusercontent.com\/slidesz\/AGV_vUdHl7VUQH4WKEAilEDMJt21shw-liTP5Slikbft35EBSyLbvWUVCNHwnCwQbX79jmfOEYu1HfPgBHOnAP3ZtkWJprPqjYSVKWVQS8rmN8U3BfzZRRRwYkOHYBOk2iTvxabhJaIi=s2048?key=MhxsOoUwM9UCIcXKZDmfhW2h","datePublished":"2025-05-06T03:44:03+00:00","dateModified":"2025-05-08T05:16:15+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/metrics\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/metrics\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/metrics\/#primaryimage","url":"https:\/\/lh7-rt.googleusercontent.com\/slidesz\/AGV_vUdHl7VUQH4WKEAilEDMJt21shw-liTP5Slikbft35EBSyLbvWUVCNHwnCwQbX79jmfOEYu1HfPgBHOnAP3ZtkWJprPqjYSVKWVQS8rmN8U3BfzZRRRwYkOHYBOk2iTvxabhJaIi=s2048?key=MhxsOoUwM9UCIcXKZDmfhW2h","contentUrl":"https:\/\/lh7-rt.googleusercontent.com\/slidesz\/AGV_vUdHl7VUQH4WKEAilEDMJt21shw-liTP5Slikbft35EBSyLbvWUVCNHwnCwQbX79jmfOEYu1HfPgBHOnAP3ZtkWJprPqjYSVKWVQS8rmN8U3BfzZRRRwYkOHYBOk2iTvxabhJaIi=s2048?key=MhxsOoUwM9UCIcXKZDmfhW2h"},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/metrics\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/"},{"@type":"ListItem","position":2,"name":"Evaluation Metrics"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/","name":"LVLM Multi-image VQA Dehallucination","description":"Authors: Dongyu Yao, Keling Yao, Harry Chi; Advisor: Prof. Katia Sycara and Dr. Yaqi Xie","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/pages\/5","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/users\/230"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/comments?post=5"}],"version-history":[{"count":13,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/pages\/5\/revisions"}],"predecessor-version":[{"id":74,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/pages\/5\/revisions\/74"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team15\/wp-json\/wp\/v2\/media?parent=5"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}