4D Reconstruction From Monocular Video

Authors: Manan Shah. Project Advisors: Yehonathan Litman, Xiaoxuan Ma, Nicolas Ugrinovic, Kris Kitani, Fernando De La Torre, Shubham Tulsiani
Results
Qualitative Results
We qualitatively evaluate the 4D reconstruction on in-the-wild videos.





It can be seen that our method is able to reconstruct the regions that are occluded and can model deformations to a great extent, as in the case of the book, where the template gaussians are for the closed book, and the frames with the book open also get modelled perfectly.
Quantitative
We further evaluate our method on the Consistent4D dataset of synthetic objects.

Ablation Study

The above table suggests the significance of each component and stage in the overall optimization objective.
Future Work
- The final reconstructions depend on the choice of hyperparameters and are prone to the failure modes of SAM3D.
- We are currently trying to improve the stability and jitter issues present in the final reconstructions.
- We are also working towards improving the initial geometry prediction stage, which currently uses SAM3D and aspire to obtain temporally consistent predictions natively, given a video.