Everyone has heard the famous tale of Isaac Newton discovering gravity from an apple falling on his head. His discovery highlights human ability to infer and generalize physical properties, even at a young age. Despite this being a basic human capability, AI struggles with physically accurate simulations and generations. Video diffusion models may output great visual results, but these may not necessarily be physically accurate, as the physics is hallucinated. On the other hand, expert-tuned models for physics simulations may not match visual expectations and require expert-crafted formulations and algorithms. We propose an easy-to-use hybrid approach that includes both learning-based and math-based methods.
Problem Statement

Given a dynamic object in motion, we want to recover its 3D geometric reconstruction and physical properties. Doing so allows our method to predict future frames of motion of the same object. It also allows the prediction of a new trajectory from different starting conditions.

Previous methods, such as PAC-NeRF [1], Gaussian Informed Continuum [4], and NeuMA [3], require the object’s material to be identified before training. Other methods like Spring-Gaus [2] automatically assume an elastic object. However, in a practical use case, the user may not know the material type ahead of time. This motivates our work, which introduces a Material-Agnostic System Identification.
Additionally, previous methods train and test on one video sequence. We believe using multiple trajectories of the same object in motion will validate our method and provide better generalization.
References
[1] Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for geometry-agnostic system identification. arXiv preprint arXiv:2303.05512, 2023
[2] Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. In European Conference on Computer Vision, pages 407–423. Springer, 2025
[3] Junyi Cao, Shanyan Guan, Yanhao Ge, Wei Li, Xiaokang Yang, and Chao Ma. Neuma: Neural material adaptor for visual grounding of intrinsic dynamics. Advances in Neural Information Processing Systems, 37:65643–65669, 2025.
[4] Junhao Cai, Yuji Yang, Weihao Yuan, Yisheng He, Zilong Dong, Liefeng Bo, Hui Cheng, and Qifeng Chen. Gic: Gaussian-informed continuum for physical property identification and simulation. arXiv preprint arXiv:2406.14927, 2024.