Methodology

Problem Formulation

Given RGB and thermal (IR) imagery from UAVs, our goal is to perform fine-grained, hierarchical triage classification over a diverse set of medical indicators. The output consists of 50+ structured labels, organized into observables (what is visually detectable) and assessments (clinical conclusions derived from those observations). Here are the sample labels : 

{
    "sev_hemorrhage_observables": {
        "blood_pooling_observed": false,
        "extensive_blood_on_body": false
    },
    "sev_hemorrhage_assessment": {
        "hemorrhage_assessed_absent": false,
        "hemorrhage_not_assessable": false
    },
    "respiratory_distress_observables": {
        "tripod_position_observed": false,
        "abnormal_head_neck_posture": false,
        "cyanosis_observed": false
    },
    "respiratory_distress_assessment": {
        "respiratory_distress_assessed_absent": false,
        "respiratory_distress_not_assessable": false
    },
    "trauma_head_neck_observables": {
        "wound_head_neck_observed": false
    },
    "trauma_head_neck_assessment": {
        "head_neck_assessed_no_wound": false,
        "head_neck_not_assessable": false
    },
    "trauma_torso_observables": {
        "wound_torso_observed": false
    },
    "trauma_torso_assessment": {
        "torso_assessed_no_wound": false,
        "torso_not_assessable": false
    },
    "trauma_upper_extremities_observables": {
        "wound_upper_extremity_observed": false,
        "amputation_upper_extremity_observed": false
    },
    "trauma_upper_extremities_assessment": {
        "upper_extremities_assessed_no_wound": false,
        "upper_extremities_not_assessable": false
    },
    "trauma_lower_extremities_observables": {
        "wound_lower_extremity_observed": false,
        "amputation_lower_extremity_observed": false
    },
    "trauma_lower_extremities_assessment": {
        "lower_extremities_assessed_no_wound": false,
        "lower_extremities_not_assessable": false
    },
    "alertness_motor_posture_observables": {
        "ambulatory": false,
        "standing_unsupported": false,
        "sitting_unsupported": false,
        "sitting_supported": false,
        "lying_down": false
    },
    "alertness_motor_posture_assessment": {
        "posture_not_assessable": false
    },
    "casualty_treatment_status_observables": {
        "tourniquet_applied_observed": false,
        "bandaging_applied_observed": false,
        "medic_actively_treating": false
    },
    "casualty_treatment_status_assessment": {
        "treatment_assessed_absent": false,
        "treatment_not_assessable": false
    },
    "physiological_state_additional_indicators": {
        "protective_hand_placement_observed": false
    },
    "subject_type": {
        "subject_is_casualty_human_actor": false,
        "subject_is_casualty_manikin": false,
        "subject_is_medic": false,
        "subject_is_robot": false
    },
    "context": {
        "close_inspection_achieved": false
    }
}

This hierarchical structure reflects real-world triage: the system must first detect visual evidence (e.g., blood pooling, posture, wounds) and then reason about medical conditions (e.g., hemorrhage, respiratory distress), often under partial visibility and ambiguity.

Design Choice: Unified Multimodal VLM

Instead of training separate specialized models for each label group, we adopt a single Vision-Language Model (VLM) that jointly predicts all outputs from multimodal inputs.

Rationale:

  • Data constraints: Limited labeled data per category makes training separate models impractical
  • Generalization: VLMs can leverage shared representations across tasks
  • Efficiency: A single model reduces deployment overhead (memory, bandwidth)
  • Reasoning capability: Joint modeling enables cross-label dependencies (e.g., posture informing respiratory distress)

The VLM takes as input:

  • RGB images / video frames
  • Thermal (IR) images
  • Additional human anatomy grounding signals (described later)

and outputs the full structured label set.

Two-Stage Pipeline Overview

A major challenge is the scarcity of paired RGB–IR data. While we have abundant RGB imagery, only a limited set (~9K images) contains aligned thermal data.

To address this, we design a two-stage pipeline:

Stage 1: RGB → IR Generation (Data Expansion)

Train a conditional diffusion model to synthesize thermal images from RGB inputs.

Stage 2: Multimodal VLM Fine-tuning

Use both real and generated RGB–IR pairs to train the triage model.

This allows us to scale training data without requiring costly thermal annotations.

Stage 1: Conditional Diffusion for RGB–IR Generation

We train a conditional latent diffusion model (LDM) to generate thermal images given RGB inputs taking inspiration from F-ViTA and TherA.
Our baseline results on F-ViTA’s checkpoints can be seen in the results and experiments section.

Conditioning Strategy

We incorporate both global and object-level conditioning:

  • Global conditioning:
    • RGB image
    • Scene-level textual description
  • Object-level conditioning:
    • Segmentation masks
    • Object-specific descriptions (including temperature cues)
    • Human Anatomy grounding outputs

This enables the model to learn fine-grained thermal behavior per object, rather than producing globally uniform outputs.

Data Augmentation via Temperature Simulation

To improve robustness under limited thermal data, we augment each IR image by generating multiple variants through object-level temperature transformations. For selected regions (e.g., wounds or body parts), we create both warmer and cooler versions, producing ~6 augmented samples per image. The key idea is to simulate physiologically plausible temperature variations rather than applying naïve pixel edits, ensuring realism while increasing diversity in the dataset.

Given an IR image and a binary mask, we estimate global and region-specific temperature ranges using pixel percentiles, constrained by human physiological limits. We then apply an adaptive multiplicative scaling (~15%) to the masked region—calibrated to reflect realistic temperature differences (≈2–5°C)—while preserving the relative intensity structure. This ensures that augmented images remain physically consistent and visually coherent, improving the generalization of the downstream diffusion model.

Stage 2: Multimodal VLM Fine-tuning

Using both real and synthesized RGB–IR pairs, we fine-tune a VLM for triage prediction.

We explore Qwen3.5 and Cosmos Reason 2 as these models provide strong multimodal reasoning capabilities and can scale to structured outputs.

Incorporating Spatial Grounding

A key challenge in triage is localization—the model must know where to look before making predictions.

To address this, we use Molmo2 for human anatomy grounding. These groundings are provided as additional inputs to the VLM, enabling:

  • Better alignment between visual evidence and predictions
  • Reduced hallucinations
  • Improved performance on spatially sensitive labels (e.g., amputations, wounds)

Training Strategy

We combine efficient fine-tuning with reasoning-aware optimization:

  • LoRA (Low-Rank Adaptation):
    Enables parameter-efficient training under hardware constraints
  • GRPO-based reinforcement learning:
    Optimizes model outputs using rewards for:
    • Prediction accuracy
    • Reasoning quality
    • Output consistency

This encourages the model to not only predict correctly, but also reason coherently across hierarchical labels.

Open Challenges

While the pipeline is effective, several challenges remain:

  • Multiview integration: Leveraging multiple camera angles for improved robustness
  • Temporal reasoning: Extending from images to videos for dynamic signals (e.g., movement, breathing)
  • Model efficiency trade-offs: Exploring whether smaller specialized models can complement the unified VLM