Pipeline
Our pipeline is shown above. Given a swing video clip, we first use a detection model to get the bat detection boxes, and then we use our keyframe detection module to get the keyframe detection. Finally, we apply a line fitting and a vanishing point detection algorithm to estimate and correct the hitting angle.
Detection Model
Detection results are pivotal in both keyframe detection and angle estimation phases of our project. We have selected the Detic model for detecting baseball bats, owing to its exceptional performance and adaptability. Detic offers comprehensive coverage across a wide range of classes within diverse datasets, which is crucial for accurately identifying specialized sports equipment such as baseball bats. Moreover, the model excels in reliably detecting and tracking baseball bats under various video conditions, ranging from professional broadcast footage to amateur recordings.
Keyframe Detection
Details of our keyframe detection module is shown above. We began by identifying keyframe candidates based on bat detection confidence scores. Since the swing motion is orthogonal to the camera axis near the keyframe, detections around this point generally exhibit the highest confidence scores. Thus, we selected the frame with the highest confidence score and its neighboring frames as keyframe candidates. We then concatenated the detection box features from the candidate frames with those from their surrounding frames, incorporating motion information into the analysis. Finally, we utilized a multi-layer perceptron (MLP) to output keyframe confidence scores and selected the frame with the highest score as the predicted keyframe.
Hitting angle estimation and correction
After detecting the bat in the keyframe, we apply PCA to the bat mask to determine its orientation. However, many broadcast videos are not filmed directly behind the pitcher’s mound and are often slightly rotated and tilted. To correct the angle, we use a vanishing point detection model.
The standard coordinate system is defined as below.
Suppose we define the vanishing vector corresponding to the x,y,z axes to be vx, vy, vz, and bat vector to be m. The following equation can be used to obtain the corrected hitting angle (the projected hitting angle inside the strike zone plane).
