Related Works

Non-rigging Methods

Current non-rigging video-to-4D methods, either per-object optimization methods or feed-forward methods [1],[2], solve this task by optimizing/predicting a canonical mesh (M)’s deformation at each timestep ({D_t}), so the rendered video aligns with the input. They start by generating a canonical mesh from a chosen canonical frame (I) via off-the-shelf image-to-3D models. Some feed-forward methods [1] further leverage 3D priors for deformation prediction. However, these methods fail to handle occlusions, extreme motion, or drastic shape/topology changes over time due to the insufficient information provided by the canonical image and limitations of an image-to-3D model. We formulate this kind of failure as
M* != Imageto3D(I*), M: canon. mesh, I: frame, *: optimal.

To resolve this problem, we proposed to learn an optimal canonical mesh M* instead of simply using image-to-3D for canonical mesh generation.

ActionMesh

Motion324

Rigging-based Methods

Current rigging-based methods approach the problem by automatically generating skeletal rigs and predicting skinning weights for 3D characters. Puppeteer [3] proposes an auto-regressive transformer for skeleton joint prediction with hierarchical ordering, combined with a topology-aware attention mechanism that encodes inter-joint relationships via skeletal graph distance. Animation is further optimized based on text-generated videos.

However, these methods lack interactive editing capabilities, they do not support user-guided skeleton adjustment or editing-driven rig generation, making it difficult to refine or adapt the pipeline for specific characters or dynamic real-world objects.

References

[1] R. Sabathier, D. Novotny, N. J. Mitra, and T. Monnier, “ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion,” 2026, arXiv:2601.16148 [cs.CV]

[2] H. Chen, X. Chen, Y. Zhang, Z. Xu, and A. Chen, “Motion 3-to-4: 3D Motion Reconstruction for 4D Synthesis,” 2026, arXiv:2601.14253 [cs.CV]

[3] Song, Chaoyue, Xiu Li, Fan Yang, Zhongcong Xu, Jiacheng Wei, Fayao Liu, Jiashi Feng, Guosheng Lin, and Jianfeng Zhang. “Puppeteer: Rig and animate your 3d models.” 2025 arXiv:2508.10898