Overview

Motivation

Generating animated 3D assets from monocular video is a fundamental challenge in computer vision with broad applications in AR/VR, films, games, and simulations. Given only a monocular video as input, the goal is to produce a canonical mesh with per-frame deformations.

Problem Statement

Current methods face two major challenges:

Non-Rigging Methods: More freedom in animation, but struggle to handle occlusions and drastic topology or shape changes over time.

Rigging-Based Methods: Have difficulty keeping the skeleton and skinning stable when shape and appearance change non-rigidly. Do not support user-guided skeleton adjustment or editing-driven rig generation, making it difficult to refine or adapt the pipeline.

Our Goal

We propose two pipelines: a non-rigging pipeline that learns an optimal canonical mesh M* that correctly deforms at each timestep, and a rigging pipeline that enables interactive skeleton editing supporting MOVE, REMOVE, and ADD operations for skeleton-driven animation from monocular video.