The Shift to 3D Robotics Traditional robotic manipulation has long relied on 2D Vision-Language-Action (VLA) models which, while capable of broad generalization, often lack the explicit 3D inductive bias required for high-precision manipulation. By incorporating spatial coordinates directly into the model’s internal map, 3D VLAs (like 3DDA and 3DFA) can ground actions in the physical geometry of a scene, leading to significantly better task performance.

The Calibration Crisis However, 3D policies introduce a critical vulnerability: they depend on perfect camera extrinsics. In real-world environments, camera mounts suffer from physical drift, vibrations, and sensor remounting. Most large-scale robotic datasets, including DROID and Open X-Embodiment, contain significant calibration noise or lack 3D annotations entirely. When a camera’s yaw is off by even a few degrees, the resulting 3D point cloud misaligns with the robot’s physical frame, leading to catastrophic grasp failures as the model attempts to interact with “ghost” objects.

Project Objective Our project aims to bridge this divide by enabling 3D policies to learn from imperfect data. We develop a system capable of implicit self-calibration during the forward pass, allowing robots to “re-center” their internal coordinate frames without the need for manual recalibration or external targets like checkerboards.