Given RGB and thermal (IR) imagery from UAVs, our goal is to perform fine-grained, hierarchical triage classification<\/strong> over a diverse set of medical indicators. The output consists of 50+ structured labels<\/strong>, organized into observables<\/em> (what is visually detectable) and assessments<\/em> (clinical conclusions derived from those observations). Here are the sample labels : <\/p>\n\n\n\n
This hierarchical structure reflects real-world triage: the system must first detect visual evidence<\/strong> (e.g., blood pooling, posture, wounds) and then reason about medical conditions<\/strong> (e.g., hemorrhage, respiratory distress), often under partial visibility and ambiguity.<\/p>\n\n\n\n\n\n\n\n
Instead of training separate specialized models for each label group, we adopt a single Vision-Language Model (VLM)<\/strong> that jointly predicts all outputs from multimodal inputs.<\/p>\n\n\n\n
Rationale:<\/strong><\/p>\n\n\n\n
\n
Data constraints:<\/strong> Limited labeled data per category makes training separate models impractical<\/li>\n\n\n\n
Generalization:<\/strong> VLMs can leverage shared representations across tasks<\/li>\n\n\n\n
Efficiency:<\/strong> A single model reduces deployment overhead (memory, bandwidth)<\/li>\n\n\n\n
Additional human anatomy grounding signals (described later)<\/li>\n<\/ul>\n\n\n\n
and outputs the full structured label set.<\/p>\n\n\n\n
\n\n\n\n
Two-Stage Pipeline Overview<\/h2>\n\n\n\n
A major challenge is the scarcity of paired RGB\u2013IR data<\/strong>. While we have abundant RGB imagery, only a limited set (~9K images) contains aligned thermal data.<\/p>\n\n\n\n<\/figure>\n\n\n\n
To address this, we design a two-stage pipeline<\/strong>:<\/p>\n\n\n\n
Stage 1: RGB \u2192 IR Generation (Data Expansion)<\/h3>\n\n\n\n
Train a conditional diffusion model to synthesize thermal images from RGB inputs.<\/p>\n\n\n\n
Stage 2: Multimodal VLM Fine-tuning<\/h3>\n\n\n\n
Use both real and generated RGB\u2013IR pairs to train the triage model.<\/p>\n\n\n\n
This allows us to scale training data without requiring costly thermal annotations<\/strong>.<\/p>\n\n\n\n\n\n\n\n
Stage 1: Conditional Diffusion for RGB\u2013IR Generation<\/h2>\n\n\n\n
We train a conditional latent diffusion model (LDM)<\/strong> to generate thermal images given RGB inputs taking inspiration from F-ViTA and TherA. Our baseline results on F-ViTA’s checkpoints can be seen in the results and experiments section.<\/p>\n\n\n\n
Conditioning Strategy<\/h3>\n\n\n\n
We incorporate both global<\/strong> and object-level conditioning<\/strong>:<\/p>\n\n\n\n<\/figure>\n\n\n\n
Object-specific descriptions (including temperature cues)<\/li>\n\n\n\n
Human Anatomy grounding outputs<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
This enables the model to learn fine-grained thermal behavior per object<\/strong>, rather than producing globally uniform outputs.<\/p>\n\n\n\n
Data Augmentation via Temperature Simulation<\/h3>\n\n\n\n<\/figure>\n\n\n\n
To improve robustness under limited thermal data, we augment each IR image by generating multiple variants through object-level temperature transformations. For selected regions (e.g., wounds or body parts), we create both warmer and cooler versions, producing ~6 augmented samples per image. The key idea is to simulate physiologically plausible<\/em> temperature variations rather than applying na\u00efve pixel edits, ensuring realism while increasing diversity in the dataset.<\/p>\n\n\n\n
Given an IR image and a binary mask, we estimate global and region-specific temperature ranges using pixel percentiles, constrained by human physiological limits. We then apply an adaptive multiplicative scaling (~15%) to the masked region\u2014calibrated to reflect realistic temperature differences (\u22482\u20135\u00b0C)\u2014while preserving the relative intensity structure. This ensures that augmented images remain physically consistent and visually coherent, improving the generalization of the downstream diffusion model.<\/p>\n\n\n\n
\n\n\n\n
Stage 2: Multimodal VLM Fine-tuning<\/h2>\n\n\n\n
Using both real and synthesized RGB\u2013IR pairs, we fine-tune a VLM for triage prediction.<\/p>\n\n\n\n
We explore Qwen3.5 and Cosmos Reason 2 as these models provide strong multimodal reasoning capabilities and can scale to structured outputs.<\/p>\n\n\n\n
Incorporating Spatial Grounding<\/h3>\n\n\n\n
A key challenge in triage is localization<\/strong>\u2014the model must know where to look<\/em> before making predictions.<\/p>\n\n\n\n
To address this, we use Molmo2<\/strong> for human anatomy grounding. These groundings are provided as additional inputs to the VLM, enabling:<\/p>\n\n\n\n
\n
Better alignment between visual evidence and predictions<\/li>\n\n\n\n
Reduced hallucinations<\/li>\n\n\n\n
Improved performance on spatially sensitive labels (e.g., amputations, wounds)<\/li>\n<\/ul>\n\n\n\n
Training Strategy<\/h3>\n\n\n\n
We combine efficient fine-tuning with reasoning-aware optimization:<\/p>\n\n\n\n
\n
LoRA (Low-Rank Adaptation):<\/strong> Enables parameter-efficient training under hardware constraints<\/li>\n\n\n\n
GRPO-based reinforcement learning:<\/strong> Optimizes model outputs using rewards for:\n
Problem Formulation Given RGB and thermal (IR) imagery from UAVs, our goal is to perform fine-grained, hierarchical triage classification over a diverse set of medical indicators. The output consists of 50+ structured labels, organized into observables (what is visually detectable) and assessments (clinical conclusions derived from those observations). Here are the sample labels : This … <\/p>\n
![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf21\/wp-content\/uploads\/sites\/163\/2026\/05\/image-14.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf21\/wp-content\/uploads\/sites\/163\/2026\/05\/image-15.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf21\/wp-content\/uploads\/sites\/163\/2026\/05\/image-16-1024x521.png\")
<\/figure>\n\n\n\n