Control via Demonstration - Computer Vision for Cinematographic Motion Control

Fig. 1 – Control via Demonstration Overview

Figure 1 lays out the control via demonstration pipeline. We used ORB SLAM3 [1] to obtain the camera trajectory from the phone inputs: RGB Video, IMU position data, and the Lidar depth map. This is then passed through the trajectory refinement layer as described in the methodology page to obtain the final output trajectory.

Results

Fig. 2 – Successfully Extracted Phone Trajectory

As seen in the video on the left, we were able to successfully extract the trajectory of the camera using the RGBD and IMU inputs.

Once we have the phone odometry, we run our inverse kinematics module to solve for the rig control. The module successfully exports these controls as we can see in the video on the right.

Fig. 3 – iPhone trajectory is converted to robot joint angles via inverse kinematics.

References

[1] Campos, C., Elvira, R., Rodriguez, J. J. G., Montiel, J. M. M., & Tardos, J. D. (2021). ORB-SLAM3: An accurate open-source library for visual, visual–inertial, and multimap SLAM. IEEE Transactions on Robotics, 37(6), 1874–1890. https://doi.org/10.1109/TRO.2021.3075644