We propose a grounded vision-language pipeline that converts raw warehouse images into structured pallet attributes and an image-level safety decision. The overall flow is: pixels → objects → semantics → decisions. First, traditional computer vision localizes candidate pallet regions. Then, a vision-language model reasons over cropped pallet instances to extract semantic attributes such as load presence and damage. Finally, instance-level predictions are aggregated into a scene-level anomaly decision.
We first use Grounding DINO to detect and localize candidate pallet regions in each warehouse image. This step is necessary because raw warehouse scenes are often cluttered, wide-angle, and contain multiple pallets at different scales. The detector outputs bounding boxes around likely pallet instances. Low-confidence detections are filtered out, duplicate or overlapping boxes are removed, and bounding boxes are slightly expanded before cropping to preserve visual context.
Every crop that is detected is then entered into a vision-language model, which helps derive pallet features in a structured format. The vision-language model predicts the presence of the pallet and the presence of the load, identifies the type of the pallet, the presence of damage on the pallet, and the presence of damage on the load. Rather than using a single prompt for all the attributes, we separately query each attribute.
Since a single warehouse image may consist of several pallets, individual predictions are made and aggregated to an overall image prediction. An anomaly is present in an image if at least one of the pallets or loads is predicted to be damaged. Such an approach is adopted due to the safety aspect of this task — one damaged pallet makes the situation hazardous.