Method

Non-rigging

Our method aims to learn an optimal canonical mesh M* that could correctly deform at each timestep. We achieve this by optimizing the vertex positions of the canonical mesh to minimize both geometric loss and photometric loss. We use the frozen ActionMesh’s deformation decoder to predict per-timestep deformation. For geometric loss, we first generate a sequence of temporally consistent independent meshes, then compute the average chamfer distance between the deformed canonical mesh and the independent mesh at each timestep. For photometric loss, we compute the average L1 loss between the rasterized images and the input video.

For independent mesh generation, we implement a stochastic latent propagation strategy inspired by SDEdit [4], where the latent from timestep t is perturbed with Gaussian noise and then denoised to form the initialization for timestep t+1 & t-1.

Rigging

We propose a pipeline that adapt the skeleton generation pipeline from puppeteer[5], which uses auto-regressive transformer to generation skeleton taking object mesh as input. Then we’ll use a user editing module which takes in user’s operation (ADD, MOVE, REMOVE), generate modified skeleton with refinement transformer.

For the first semester, we tried deterministic skeleton editing method, and we will do non-deterministic pipeline or use the editing to guide the skeleton generation.

References

[4] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon: SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. ICLR 2022

[5] Song, Chaoyue, Xiu Li, Fan Yang, Zhongcong Xu, Jiacheng Wei, Fayao Liu, Jiashi Feng, Guosheng Lin, and Jianfeng Zhang. “Puppeteer: Rig and animate your 3d models.” 2025 arXiv:2508.10898 (2025)