RGB to IR generation baseline results on F-ViTA checkpoints –
As seen above, there is a gap between the predicted IR images and the gorund truth IR images. The images from NIRSCENE checkpoint are the closest to the ground truth but even they seem to have just learned how to create a grayscale image. On comparing column 2 and 3, we can see that NIRSCENE predicted image does not follow the rules of physics, for example a patch under the shade of the trees would be cooler, a body lying in the sun would be uniformly hotter etc. We can improve this by finetuning further and using temperature cues as additional conditioning.
Improvements in RGB + IR modality fusion
As shown in the table, combining RGB and IR consistently improves performance compared to single-modality inputs. For blood pooling detection, RGB-only models tend to produce higher false positives due to visual ambiguity (e.g., smoke or lighting), while IR helps suppress these errors by providing clearer thermal cues. This is reflected in the reduced FP counts when using RGB+IR (e.g., Qwen3.5: 12 → 5).
For motor posture estimation, IR significantly improves robustness in cases where RGB fails due to poor visibility or viewpoint. While IR-only sometimes underperforms due to lack of texture, combining it with RGB yields the best overall balance—improving true positives and reducing false negatives (e.g., Qwen3.5 TP: 32 → 37, FN: 18 → 13).
Across all three models, the trend is consistent: multimodal fusion leads to fewer hallucinations, better recall in difficult cases, and more stable predictions overall. This supports our design choice of using RGB+IR as the primary input to the VLM.
Molmo2 Human Anatomy Grounding
We evaluate four grounding strategies against a no-grounding baseline. All experiments use Cosmos Reason 2 as the downstream triage model, with RGB input only.
Point coordinates overlaid
Skeleton overlaid
Baseline, no grounding at all , gets a macro F1 of 16. Adding text coordinates bumps it slightly to 17.9. But point dots, overlaying actual visual dot markers on the body parts — is the clear winner at 19.6 macro F1, with the biggest gains on Posture and Trauma.
The skeleton overlay is interesting, it actually hurts posture performance, dropping to 11.3, probably because the overlay adds visual clutter that confuses the model.
The control condition, shuffled coordinates, deliberately wrong spatial labels, actually scores highest on Hemorrhage at 39.6. That’s a signal that Hemorrhage detection is largely appearance-driven and doesn’t benefit much from spatial context. The model is essentially ignoring the grounding and relying on color cues.
The takeaway: grounding helps, but only when it’s clean and precise. Point dots work because they’re minimal, they add spatial signal without cluttering the visual context the VLM needs.