---

Introduction

One aspect of 3D scene understanding is to create visual systems that can faithfully reconstruct full 360-degree objects from monocular videos. We introduce a new method to recover “Complete 4D Reconstructions” of objects from a single, “in-the-wild” video.

The Challenge

Reconstructing moving objects from a single camera view is an open problem in computer vision. Some challenges that we aim to address:

• Deformable objects: Objects in videos may undergo large, unconstrained deformations, making it challenging for reconstruction methods based on fixed templates or rigid object assumptions to recover these deformations over time.
• Occluded objects: Objects of interest in the video may be occluded by other objects across time from a single view, making it harder to reconstruct the full object solely based on the observed video sequence.
• In-the-wild generalization: Real-world videos contain a diverse category of objects, making it harder to solely rely on a template-based approach for specific object classes such as humans or animals.

Our Approach

Our method bridges the gap between static 3D understanding and dynamic 4D motion. We utilize a state-of-the-art 3D reconstruction prior (SAM3D) and enhance it with three key innovations:

1. Causal Latent Conditioning: We propagate information across video frames to ensure the object’s shape stays consistent over time.
2. Deformable 3D Representation: We use a time-varying deformable 3D representation parametrized by sparse control nodes.
3. Occlusion-Aware Supervision: To align the deformable representation with the input video, we use a rendering-based supervision while carefully taking care of occlusions and unobserved regions in the video.