Introduction

CONTENT WARNING: This website contains images that may include graphic injury and visible blood. Viewer discretion is advised.

This project is part of Team Chiron’s entry in the DARPA Triage Challenge (DTC), a DARPA-funded program aimed at developing autonomous systems capable of assessing casualties in mass-casualty scenarios. The challenge tasks teams with building UAV-based pipelines that can detect, localize, and triage injured individuals without any human in the loop. We are currently in Year 3 of the competition. Teams are scored on the number of lives saved. Achieving a high score requires accurate multimodal perception, robust localization, and reliable clinical inference across all of these variables simultaneously. Below is a short snippet of the DTC data.

Mass casualty events are chaotic. First responders often rely on incomplete information, limited visibility, and time pressure. In triage, minutes can mean survival.

In disaster scenarios, responders need to quickly answer:

• Who is injured?
• How badly?
• Who needs help first?

But this is hard because victims could be partially hidden, injuries may not be able in standard RGB images and scenes are cluttered and dynamic.

Look at the cases below:

Our Approach

We treat triage as both a perception and reasoning problem.
Input are RGB/IR images and the outputs are 50+ labels.
Labels like severe_heamorrage, trauma_head_neck, blood_pooling_observed

Input data – RGB Images/Videos/Multiview + IR (Thermal) Images/Videos/Multiview

Output Labels (50+) – Severe Hemorrhage, Respiratory Distress, Trauma Head/Neck, Alertness Ocular

Multi-modal (RGB + IR) Fusion

Real-world UAV data introduces several challenges: casualties may be occluded (e.g., under foliage), captured from high altitudes, or partially visible due to environmental conditions such as smoke or clutter. In such cases, relying on RGB alone is often insufficient. We therefore incorporate thermal (IR) imagery as a complementary modality, allowing the model to fall back on thermal cues when RGB signals are weak or ambiguous, and to jointly reason over both modalities when available.

IR proves particularly effective in detecting hidden or low-visibility casualties. For example, casualties obscured by tree foliage may be difficult to identify in RGB images, but remain clearly visible in thermal imagery due to temperature contrast.

Similarly, in aerial views where subjects blend into the background, IR provides sharper structural cues, enabling more reliable posture estimation. We validate this using Qwen3.5: for a motor posture (lying down) case, RGB alone is “Not Assessable,” while both IR-only and RGB+IR correctly identify the posture as “Lying Down.”

In addition to improving visibility, IR also helps reduce hallucinations, particularly in challenging scenes. For instance, in the presence of smoke, RGB-only predictions may incorrectly attribute blood to the torso due to visual ambiguity. However, IR does not exhibit this confusion, as thermal signatures clearly distinguish between blood and background. When combining both modalities, the model correctly predicts no blood on the torso, demonstrating improved reliability.

Overall, multimodal fusion enables the system to be more robust, accurate, and less prone to failure in real-world scenarios. Quantitative improvements are reported in the Experiments and Results section.

Human Anatomy Grounding

Standard VLMs fail triage not because they lack medical knowledge — they fail because they don’t know where to look. Asking a model whether a leg is amputated when it can’t locate the leg is asking for a guess.

In the example above, Cosmos Reason 2 misidentifies the amputated leg and blood-pooled region as the head — a fundamental localization failure, not a reasoning one. To address this, we decompose the problem into two stages: localize first, then assess.

We use Molmo2, an open-weight VLM specialized in spatial grounding and pointing. Given a UAV frame, Molmo2 reliably localizes body parts head, torso, arms, legs — and returns precise point coordinates. These coordinates are then passed as additional spatial context to Cosmos Reason 2, giving it an explicit map of where each body part is before it makes any clinical judgment. The results of this approach are in Experiments and Results section.