Our proposed pipeline combines the three main directions from the related work: multiview human capture, contact-aware hand-object reasoning, and physics-grounded deformable object modeling.
First, we use Contact4D as the main reference for recovering a consistent parametric human model from multiview video. In our pipeline, this corresponds to estimating the body pose and detailed hand pose across time. Its multiview triangulation and refinement ideas are useful because clay shaping involves frequent hand occlusions, so single-view pose estimates will not be reliable enough.
Next, we use DyTact as inspiration for estimating hand-object contact from visual evidence. In our setting, this means identifying where the hands touch the clay and using that contact information to guide reconstruction. This contact step is important because the clay deformation is not random; it is caused by local hand pressure and motion.
Finally, we use GIC as the foundation for reconstructing the clay as a deformable, physics-grounded object. Instead of treating the clay as only a surface mesh or visual reconstruction, we want a representation that can evolve over time in a physically meaningful way. GIC’s use of dynamic 3D Gaussians, material particles, and simulation gives us a direction for modeling clay-like deformation.
Together, these components form a joint 4D capture pipeline: Contact4D provides the human pose, DyTact provides contact reasoning, and GIC provides deformable clay reconstruction. The final output is a time-varying representation containing body pose, hand pose, hand–clay contact, and physics-grounded clay geometry.
Capture Setup
Our capture setup uses a half-dome of 17 synchronized GoPro cameras for external multiview video, along with Project Aria glasses for egocentric views. Camera calibration is performed with COLMAP using stationary views and additional mobile sweep images to improve coverage. This setup is designed to capture both the global human motion and the fine local hand–clay interaction needed for reconstruction.