Comparative Decoding
Motivation: How do two sets of images differ? Understanding dataset differences is important for: analyzing datasets, diagnosing model failures, discovering hidden patterns in large visual collections.
Our Baseline: VisDiff: Describing Differences in Image Sets with Natural Language
Proposed method: We propose comparative decoding, where language generation is directly conditioned on two image groups during decoding. The method measures token importance by comparing how likely each word is under images from group A versus group B, allowing visually discriminative concepts to be emphasized during generation.
Example:
- Group 1: Oranges on a tree
- Group 2: Bird nests on a tree
- Ground Truth Difference: Tree attachment (Oranges vs Bird nests)
- Generated Hypotheses: orange trees growing oranges
Bio-lab Video Anomaly Reasoning
Motivation. Building an VLM agent framework for video anomaly detection is a promising method for AI scene understanding, which is a critical building block for lab automation.
Framework. We propose O-VAD, object centric analysis for video anomaly detection. Specifically, it contains three stages, incorporating computer vision with text generation.
- Stage 1 – Object Recognition. Aiming at identify all objects of interest in the scene, VLM is prompted to identify objects and SAM [b1] model is used for segmentation.
- Stage 2 – Object Tracking. TubeletGraph model is used for object frame-wise state graph generation, followed by VLM describing changes and interactions in detail.
- Stage 3 – Anomaly Reasoning. Given all the context gathered in Stage 1 and 2, VLM goes through CoT reasoning to output anomaly status, types, reasons and effects.
Results.
Demo.
Agentic VLMs for Biomedical Image Analysis
Motivation: Biomedical image analysis requires VLMs to do more than generate fluent descriptions. A reliable biomedical VLM must first perceive scientific imagery faithfully, and then reason over structured biological evidence using computation and tools. We study these two requirements through two projects: Mutual Colormap Dependence (MCD), which audits whether VLMs are robust to scientific image visualization choices, and CCR-Agent, which uses iterative tool-grounded reasoning for cryo-ET question answering.
Project 1 — Mutual Colormap Dependence: Perception Audit
Scientific images are often visualized using false-color colormaps such as gray, viridis, plasma, or magma. In principle, changing the colormap should not change the model’s answer because the underlying scalar image remains identical. MCD tests whether current VLMs satisfy this basic invariance property. We measure answer drift using Mutual Colormap Dependence, the conditional mutual information in nats between a finite orbit of monotonic, intensity-preserving colormaps and the model’s answer for a fixed image-question pair. Lower values indicate greater robustness, and MCD equal to zero would mean perfect invariance.
Takeaway: Current VLMs can change their answers when only the image colormap changes, revealing a perception robustness issue in biomedical imagery.
Project 2 – CCR-Agent: Tool-Grounded Spatial Reasoning for Cryo-ET
Many biomedical questions cannot be answered from a single image caption. In cryo-electron tomography, the model must reason over large particle-level spatial maps containing protein identities, coordinates, clusters, and distributions. CCR-Agent addresses this by iteratively computing spatial facts, calling specialized tools, and using the resulting evidence to answer biological questions in natural language.
Takeaway: Iterative computation over structured spatial evidence is much more effective than standard VLM-only or retrieval-based baselines for cryo-ET QA.