The pipeline described on the Methods page was tested on a synthetic data set containing 1,000 images of pallets, including images of single and multiple pallets, loaded and unloaded pallets, and images of damaged and undamaged pallets. Each detected pallet was classified using the VLM, and the prediction at the instance level was combined into an anomaly prediction at the image level. Images were flagged as anomalies if any detected pallet or load had been predicted to be damaged. “Uncertain” classifications did not count as damage.
In the zero-shot experiment, Qwen 2.5 performed better than Florence-2 in terms of precision, as it was able to detect damaged pallets with high accuracy. Nevertheless, Qwen 2.5 had low recall because it could not identify a lot of damaged pallets. On the other hand, Florence-2 exhibited a behavior that is exactly the opposite of Qwen 2.5.
Overall, Qwen was more conservative and reliable when flagging damage, while Florence-2 was more aggressive and better at catching damaged cases. This shows an important tradeoff for safety-sensitive warehouse automation: high precision reduces false alarms, but high recall is also important because missing a damaged pallet can create operational risk.
The system also provided interpretative results for each individual detection. In cases where there was a healthy loaded pallet, the VLM recognized that there is a pallet, load, and no damage. If the pallet had sustained damage, it could be recognized by the model when there were visible indications of damage to the pallet structure.
All in all, based on our findings, zero-shot VLMs can act as a useful tool for structured reasoning when detecting anomalies on pallets; yet, their reliability is not enough to be applied alone in the process of ensuring safety. Specifically, we found out that Qwen demonstrates a better precision with fewer false detections, while Florence-2 shows improved recall but has more false detections. It implies that more attention should be paid to context-aware decision making, rather than to developing additional image-level models for zero-shot reasoning. Therefore, our next step will be implementing a memory model based on the retrieval of examples that are reliable.