3D Visuomotor Imitation Learning
Recent visuomotor imitation learning methods show that 3D representations are useful for manipulation because point clouds directly encode robot-object geometry. 3D Diffusion Policy (DP3) combines point cloud observations with a diffusion-based action prediction model, making it a strong low-level execution backbone for tasks that require precise geometric control.
In this project, we use a DP3-style low-level policy, but we add explicit goal conditioning. The low-level policy receives the current scene, the observed hand configuration, the predicted goal hand configuration, and the robot state. It then predicts robot actions through a diffusion denoising process.
Goal-Conditioned Hierarchical Policies
ArticuBot demonstrates a hierarchical, goal-conditioned policy for articulated object manipulation with parallel-jaw grippers. Its high-level policy predicts end-effector goal points from the current observation, and its low-level policy executes actions conditioned on the predicted goal. This architecture is effective because it separates interaction-goal prediction from action execution.
Our project follows this high-level / low-level structure, but targets dexterous hands rather than parallel-jaw grippers. This makes the goal design more challenging. A parallel-jaw gripper can often be represented by a few gripper points, while a dexterous hand needs to represent multiple fingers, palm placement, and richer contact geometry.
Dexterous Grasp Priors
Large-scale dexterous grasp datasets provide useful priors for understanding hand-object interaction. Dexonomy provides a taxonomy of dexterous grasp types, helping organize the diversity of hand configurations. GraspXL provides large-scale grasping motions for diverse objects, offering another source of dexterous hand-object interaction data.
These datasets motivate our focus on sparse but expressive hand representations. Rather than representing a goal only as an object pose or a single robot end-effector position, we represent the desired dexterous hand configuration using 3D points on fingers, finger links, and the palm.
References
[1] Ze, Yanjie, et al. “3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations.” Proceedings of Robotics: Science and Systems, 2024. https://3d-diffusion-policy.github.io/
[2] Wang, Yufei, et al. “ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulation.” Proceedings of Robotics: Science and Systems, 2025. https://articubot.github.io/
[3] Chen, Jiayi, et al. “Dexonomy: Synthesizing All Dexterous Grasp Types in a Grasp Taxonomy.” Proceedings of Robotics: Science and Systems, 2025. https://pku-epic.github.io/Dexonomy/
[4] Zhang, Hui, et al. “GraspXL: Generating Grasping Motions for Diverse Objects at Scale.” European Conference on Computer Vision, 2024. https://eth-ait.github.io/graspxl/