{"id":2,"date":"2025-05-06T05:59:15","date_gmt":"2025-05-06T05:59:15","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/?page_id=2"},"modified":"2025-12-09T16:13:45","modified_gmt":"2025-12-09T16:13:45","slug":"sample-page","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/","title":{"rendered":"Method"},"content":{"rendered":"\n

Overall Pipeline<\/h2>\n\n\n\n

In this work, we propose a training-free pipeline<\/strong> for text-guided medical image segmentation<\/strong>. Instead of relying on manually annotated pixel-level labels or additional model fine-tuning, we require minimal manual effort, by allowing the users to provide a natural language prompt like \u2018segment the optic disc\u2019 to obtain the desired medical segmentation masks. In our pipeline, we use tunable test-time parameters<\/strong>, a grounding model <\/strong>with reasoning capabilities (CogVLM<\/strong> [3]), a segmentation model (SAM<\/strong> [1]), and a validation model <\/strong>to iteratively refine our segmentation results without needing any ground truth values with Bayesian optimization. <\/strong><\/p>\n\n\n\n

\n
\n
\"\"<\/figure>\n<\/figure>\n<\/blockquote>\n\n\n\n

Bayesian Optimization<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

We apply Bayesian Optimization[2] to search for the optimal configuration of LTAs that maximizes the validator\u2019s score. The table above displays the operations we used to help achieve the best results. <\/p>\n\n\n\n

Proxy Validation<\/h2>\n\n\n\n

We perform a proxy validation with evaluation with zero-shot classification and image-text matching. <\/p>\n\n\n\n

For pseudo evaluation with zero-shot classification<\/strong>, we start by creating a test image that keeps only the region indicated by the predicted segmentation mask. To evaluate whether this region matches the intended anatomical structure, we prompt a general LLM with a template that includes a description of the target region and a description of the full image context. The LLM outputs several contrastive text labels. These labels, together with the test image, are fed into a vision-language model such as BioMedCLIP [4]. The model performs zero-shot classification, and the probability assigned to the target description is used as the zero-shot score.<\/p>\n\n\n\n

Zero-shot classification focuses on medical terminology, but it does not check whether the segmented region actually looks correct. To capture visual characteristics such as color, shape, or texture, we add a second evaluation method based on image\u2013text matching<\/strong>. We prompt an LLM to generate descriptive sentences about how the target region is expected to appear. The same vision-language model is then used to compute the similarity between the test image and each of these descriptions. The average similarity becomes the image\u2013text matching score.<\/p>\n\n\n\n

The final validation score for a predicted mask is obtained by combining the zero-shot classification score and the image\u2013text matching score.<\/p>\n\n\n\n

Citations<\/h2>\n\n\n\n

[1] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In ICCV, 2023.<\/p>\n\n\n\n

[2] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In NeurIPS, 2012.<\/p>\n\n\n\n

[3] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. NeurIPS, 2024.<\/p>\n\n\n\n

[4] Zhang, H., Li, Y., Li, Y., Tao, C., Zhang, T., Zhang, Y., Wang, Y., Gao, P., & Chen, W. (2023). BioMedCLIP: A Vision\u2013Language Foundation Model for Biomedical Applications.<\/em> arXiv:2303.00915.<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

Overall Pipeline In this work, we propose a training-free pipeline for text-guided medical image segmentation. Instead of relying on manually annotated pixel-level labels or additional model fine-tuning, we require minimal manual effort, by allowing the users to provide a natural language prompt like \u2018segment the optic disc\u2019 to obtain the desired medical segmentation masks. In … <\/p>\n

Continue reading “Method”<\/span><\/a><\/p>\n","protected":false},"author":225,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"open","template":"","meta":{"footnotes":""},"class_list":["post-2","page","type-page","status-publish","hentry"],"yoast_head":"\nMethod - Medical Segmentation with Foundation Models: A Prompt-Based, Text-Guided, Training-Free Pipeline<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - Medical Segmentation with Foundation Models: A Prompt-Based, Text-Guided, Training-Free Pipeline\" \/>\n<meta property=\"og:description\" content=\"Overall Pipeline In this work, we propose a training-free pipeline for text-guided medical image segmentation. Instead of relying on manually annotated pixel-level labels or additional model fine-tuning, we require minimal manual effort, by allowing the users to provide a natural language prompt like \u2018segment the optic disc\u2019 to obtain the desired medical segmentation masks. In … Continue reading "Method"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/\" \/>\n<meta property=\"og:site_name\" content=\"Medical Segmentation with Foundation Models: A Prompt-Based, Text-Guided, Training-Free Pipeline\" \/>\n<meta property=\"article:modified_time\" content=\"2025-12-09T16:13:45+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-content\/uploads\/sites\/130\/2025\/12\/Screenshot-2025-12-09-104304.png\" \/>\n\t<meta property=\"og:image:width\" content=\"651\" \/>\n\t<meta property=\"og:image:height\" content=\"334\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/sample-page\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/sample-page\\\/\",\"name\":\"Method - Medical Segmentation with Foundation Models: A Prompt-Based, Text-Guided, Training-Free Pipeline\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/sample-page\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/sample-page\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/wp-content\\\/uploads\\\/sites\\\/130\\\/2025\\\/12\\\/Screenshot-2025-12-09-104304.png\",\"datePublished\":\"2025-05-06T05:59:15+00:00\",\"dateModified\":\"2025-12-09T16:13:45+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/sample-page\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/sample-page\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/sample-page\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/wp-content\\\/uploads\\\/sites\\\/130\\\/2025\\\/12\\\/Screenshot-2025-12-09-104304.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/wp-content\\\/uploads\\\/sites\\\/130\\\/2025\\\/12\\\/Screenshot-2025-12-09-104304.png\",\"width\":651,\"height\":334},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/sample-page\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/\",\"name\":\"Medical Segmentation with Foundation Models: A Prompt-Based, Text-Guided, Training-Free Pipeline\",\"description\":\"Student: Colleen Que | Collaborator: Qifeng Wu | Research Mentor: Xingjian Li | Advisor: Min Xu (CMU)\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team7-1\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - Medical Segmentation with Foundation Models: A Prompt-Based, Text-Guided, Training-Free Pipeline","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/","og_locale":"en_US","og_type":"article","og_title":"Method - Medical Segmentation with Foundation Models: A Prompt-Based, Text-Guided, Training-Free Pipeline","og_description":"Overall Pipeline In this work, we propose a training-free pipeline for text-guided medical image segmentation. Instead of relying on manually annotated pixel-level labels or additional model fine-tuning, we require minimal manual effort, by allowing the users to provide a natural language prompt like \u2018segment the optic disc\u2019 to obtain the desired medical segmentation masks. In … Continue reading \"Method\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/","og_site_name":"Medical Segmentation with Foundation Models: A Prompt-Based, Text-Guided, Training-Free Pipeline","article_modified_time":"2025-12-09T16:13:45+00:00","og_image":[{"width":651,"height":334,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-content\/uploads\/sites\/130\/2025\/12\/Screenshot-2025-12-09-104304.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/","name":"Method - Medical Segmentation with Foundation Models: A Prompt-Based, Text-Guided, Training-Free Pipeline","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-content\/uploads\/sites\/130\/2025\/12\/Screenshot-2025-12-09-104304.png","datePublished":"2025-05-06T05:59:15+00:00","dateModified":"2025-12-09T16:13:45+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-content\/uploads\/sites\/130\/2025\/12\/Screenshot-2025-12-09-104304.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-content\/uploads\/sites\/130\/2025\/12\/Screenshot-2025-12-09-104304.png","width":651,"height":334},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/sample-page\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/","name":"Medical Segmentation with Foundation Models: A Prompt-Based, Text-Guided, Training-Free Pipeline","description":"Student: Colleen Que | Collaborator: Qifeng Wu | Research Mentor: Xingjian Li | Advisor: Min Xu (CMU)","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-json\/wp\/v2\/pages\/2","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-json\/wp\/v2\/users\/225"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-json\/wp\/v2\/comments?post=2"}],"version-history":[{"count":7,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-json\/wp\/v2\/pages\/2\/revisions"}],"predecessor-version":[{"id":135,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-json\/wp\/v2\/pages\/2\/revisions\/135"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team7-1\/wp-json\/wp\/v2\/media?parent=2"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}