Our initial experiments were designed to reproduce paper results as well as to adapt pretrained or fine-tuned models to our project-specific datasets.
Segmentation
Adapting a generic YOLOv8 model to the TenebrioVision dataset yielded impressive results in-line with the authors’ (Figure 1). This outcome establishes a solid foundation upon which to run further experiments. SAM3 also produces strong results out-of-the-box (Figure 2).
Pose Estimation
We generated auto-labels for the tenebrio molitor using principal component analysis on the instance segmentation masks provided in the TenebrioVision dataset. This approach flattened the mask pixels onto a single principal component axis and allowed us to infer head and tail as the endpoints along that line. While fast and convenient, using PCA for this task has some notable limitations:
- While worms that are relatively straight will perform well (Figure 3), worms that are more contorted will perform particularly poorly (not shown)
- Once PCA collapsed mask pixels onto a line, determining the head vs. the tail is not obvious and likely a source of inconsistent labeled data that degrades results
Next Steps
We have a number of tests to run to improve and build upon the above experiments and initial results:
- Use SAM3 segmentation masks as input to pose estimation module
- Use skeletonization methods to increase head and tail accuracy. This same method could also be used to add more body points along the length of the worm, further increasing accuracy as per the findings in the DeepLabCut paper.
- Explore optical flow methods such as RAFT, CoTracker, and SpatialTracker to enable tracking through video data
- Includes robustness to worms moving in-and-out of occlusion due to other worms or substate