---

Method

Given a monocular video sequence $\mathcal{I} = \{\mathbf{I}^i\}_{i=1}^N$ with associated object masks $\mathcal{M} = \{\mathbf{M}^i\}_{i=1}^N$, our goal is to reconstruct a 4D representation, which is factorized into a canonical set of $N_\mathcal{G}$ 3D gaussians $\mathcal{G}^\star = \{\boldsymbol{g}_m^\star\}_{m=1}^{N_\mathcal{G}}$ and a sequence of per-frame deformation fields $\{D^i\}_{i=1}^N$. To efficiently represent the deformation, we parameterize $D^i$ through a sparse set of $N_p$ control nodes to deform each canonical gaussian with time-varying SE(3) transformations.To achieve this, our framework consists of three modules. First, we initialize a sequence of 3D gaussian reconstructions using an off-the-shelf image-to-3D model, while enforcing temporal consistency across frames through causal latent conditioning. Next, we use these per-frame single-view reconstructions to guide the learning of our 4D representation, where both canonical gaussians and per-frame deformation fields are jointly optimized to build a coherent deformable 3D structure. Finally, we match the reconstructed representation to the input video through occlusion-aware rendering while also incorporating generic image priors for plausible reconstruction in unobserved regions.

Causal Single-view Reconstruction

We begin by obtaining a per-frame 3D reconstruction using an off-the-shelf flow matching-based image-to-3D model (SAM3D¹). While such a model can reconstruct plausible 3D gaussians from a single image, applying them independently to each frame of a video leads to temporal inconsistencies. To alleviate this issue, we enforce temporal consistency through causal latent propagation as shown in the figure.

Deformable 3D Representation and Optimization

We factorize the 4D representation into canonical 3D gaussians $\mathcal{G}^\star$and per-frame deformation fields $\{D^i\}_{i=1}^N$ parameterized by a sparse set of control nodes similar to SC-GS². Each node is associated with a time-varying learnable transformation $[\mathbf{R}^i_k|\mathbf{t}^i_k]\in \mathrm{SE}(3)$. The per-frame initialized 3D gaussians obtained from SAM3D then guide the joint optimization of the canonical gaussians and the deformation field.

Optimization Objective

The overall optimization objective to guide the canonical gaussians and deformation field is given by $\mathcal{L}_\text{rec}$, which is further decomposed as follows,

Occlusion-aware Rendering Supervision and Reconstruction

While the deformable 3D optimization provides a temporally coherent 4D representation across frames, it does not directly enforce alignment with the appearance observed in the input video. Moreover, complex occlusions and incomplete observations commonly present in in-the-wild videos make it difficult to rely solely on simple rendering supervision.

To address this, we introduce an occlusion-aware appearance reconstruction framework that preserves observed image details while maintaining plausible content in unseen regions. The occlusion mask and ground truth reference are constructed as follows:

Using the above method to generate the ground truth, we use the following loss for optimization at this stage:

---

References

1. Team, S.D., Chen, X., Chu, F.J., Gleize, P., Liang, K.J., Sax, A., Tang, H., Wang, W., Guo, M., Hardin, T., Li, X., Lin, A., Liu, J., Ma, Z., Sagar, A., Song, B., Wang, X., Yang, J., Zhang, B., Dollár, P., Gkioxari, G., Feiszli, M., Malik, J.: Sam 3d: 3dfy anything in images. arXiv (2025) ↩︎
2. Huang, Y.H., Sun, Y.T., Yang, Z., Lyu, X., Cao, Y.P., Qi, X.: Sc-gs: Sparsecontrolled gaussian splatting for editable dynamic scenes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2024) ↩︎