Architecture Overview
Our framework consists of two stages. First, a high-level policy predicts a dexterous 3D sub-goal from the current scene and observed hand state. Second, a low-level goal-conditioned diffusion policy executes actions using the predicted sub-goal as guidance.
High-Level Policy: GMM-Based Dexterous Goal Prediction
ArticuBot uses a weighted displacement model to predict approximate end-effector goal points. In our framework, we use a Gaussian Mixture Model (GMM) mode for the high-level policy. The model predicts a distribution over possible goal displacements, allowing it to represent uncertainty in the target dexterous hand configuration.
The high-level model takes the scene point cloud and observed end-effector points as input. It predicts the target 3D hand goal, represented as sparse points on the dexterous hand. This predicted goal is then passed to the low-level policy as explicit geometric guidance.
Low-Level Policy: Goal-Conditioned 3D Diffusion Policy
The low-level policy is a modified DP3-style diffusion policy. It conditions action denoising on three types of 3D information: the scene point cloud, the observed hand points, and the predicted goal hand points. It also conditions on the robot state.
To make goal conditioning effective, the encoder separately reasons about the relationship between the scene and the observed hand, and the relationship between the scene and the goal hand. This allows the policy to compare where the hand is now with where it should move next.
3D Goal Representations for Dexterous Hands
A key contribution of this project is the design of 3D goal representations for dexterous hands. The goal representation should be compact enough for the high-level policy to predict, but expressive enough to guide the low-level policy through contact-rich manipulation.
To study this trade-off, we compare three hand-geometry variants: a 4-point goal, which uses one 3D point per finger to compactly capture target finger positions; a 5-point goal, which adds one palm point to provide global hand pose guidance while remaining compact; and a 12-point goal, which places one 3D point on each finger link to represent richer finger articulation and hand shape.
We also study temporal variants of the goal. A static sub-goal represents the hand at a single key interaction frame, while a multi-frame sub-goal represents a short grasp-phase trajectory. The multi-frame version provides additional guidance during the transition from approach to stable contact.
