Overview
We propose a fully automated computer vision pipeline for discovering behavioral tells in baseball pitchers from broadcast video. Given a set of labeled delivery clips, the system extracts visual features from pre-release keyframes and applies supervised probing to test whether any latent signal predicts pitch type or location. The pipeline requires no manual annotation beyond pitch outcome labels and is designed to generalize across pitchers.
Keyframe Extraction
Tell signals, if present, are most likely to appear during the pre-release phase of the delivery. We define the maximum knee lift as our target keyframe, as it precedes ball release while the pitcher’s hands remain in a potentially informative configuration. MLS is localized automatically by extracting the vertical position of the lead knee keypoint over time, applying Gaussian smoothing to reduce jitter, and identifying the local minimum corresponding to peak knee lift. All subsequent processing operates on this single keyframe per clip.
Feature Extraction and Tell Detection
We extract a variety of dense visual features from each glove patch using various vision models such as DINOv2, SAM 3D Body, and SAM3, producing a high-dimensional representation that captures fine-grained appearance. A linear probe is trained on these features with pitch type or location as the target label. Macro F1 and balanced accuracy are used as the primary evaluation metrics to account for class imbalance across pitch types.