Pipeline
Our pipeline is shown above. Given a swing video clip, we first use a keyframe prediction module to estimate the keyframe, and then we use our detection module to detect the baseball bat. Finally, we apply a line fitting and a vanishing point detection algorithm to estimate and correct the hitting angle.
Keyframe Prediction
Details of our keyframe detection module are shown above. Given an untrimmed video, we apply a coarse-to-fine approach to predict the hitting moment. First, we identify the swing action snippets within the untrimmed video, which typically correspond to a 2-3 second clip. Next, we extract image features using DINOv2 for each frame within the predicted snippet. Finally, a feed-forward layer generates keyframe confidence scores, and the frame with the highest score is selected as the keyframe
Detection Model
Detection results are pivotal in both keyframe detection and angle estimation phases of our project. We have selected the Detic model for detecting baseball bats, owing to its exceptional performance and adaptability. Detic offers comprehensive coverage across a wide range of classes within diverse datasets, which is crucial for accurately identifying specialized sports equipment such as baseball bats. Moreover, the model excels in reliably detecting and tracking baseball bats under various video conditions, ranging from professional broadcast footage to amateur recordings.
Hitting angle estimation and correction
After detecting the bat in the keyframe, we apply PCA to the bat mask to determine its orientation. However, many broadcast videos are not filmed directly behind the pitcher’s mound and are often slightly rotated and tilted. To correct the angle, we use a vanishing point detection model.
The standard coordinate system is defined as below.
Suppose we define the vanishing vector corresponding to the x,y,z axes to be vx, vy, vz, and bat vector to be m. The following equation can be used to obtain the corrected hitting angle (the projected hitting angle inside the strike zone plane).
