Motivation
Clay shaping is a contact-rich manipulation task where small hand and finger motions directly change the shape of the material over time. A normal video captures appearance, but it misses the underlying process: how the hands move, where contact occurs, and how the clay geometry evolves. Our goal is to build a capture system that can recover this full 4D process, supporting applications in digital preservation, skill analysis, and interactive learning.
Problem Summary
Given synchronized multiview RGB video and contact sensing, we aim to recover a temporally consistent 4D representation of a person shaping clay. This includes the body pose, detailed hand pose, hand–clay contact, and physics-grounded deformable clay geometry. The key difficulty is that these outputs are strongly coupled: hand motion causes contact, contact drives deformation, and deformation changes the visible geometry over time. In this project, we aim to recover body pose, hand pose, hand–clay contact, and physics-grounded clay geometry from multiview RGB video.
Challenges
This problem is difficult because hands and clay frequently occlude each other, especially during fine manipulation. Clay is also soft and deformable, so its shape changes continuously rather than behaving like a rigid object. In addition, pottery involves precise and sometimes interlaced finger motions that are hard to observe from cameras alone. A practical system also needs to remain physically plausible while avoiding a capture setup that is too invasive or difficult to deploy.