Robotic manipulation is increasingly driven by Vision–Language–Action (VLA) models, which connect perception, language, and action. However, these models are difficult to scale because real-world robot data is expensive to collect, time-consuming to annotate, and often unavailable to the broader community. As a result, VLA models can overfit to limited training environments and require additional fine-tuning when deployed in new settings.
This project explores simulation as a scalable alternative. We focus on reconstructing real-world scenes into simulation-ready 3D environments, aligning them with robot and camera viewpoints, and using them as a foundation for synthetic data generation, benchmarking, reward learning, and future VLA training.
Why Simulation
Challenges in Robot Learning
Real-world teleoperation is expensive and difficult to scale. Large-scale robot datasets are often not publicly accessible, and VLA models can overfit to narrow training distributions. As a result, deploying policies in new environments often requires significant additional adaptation.
Our Direction
We investigate a simulation-centered pipeline that reconstructs real-world scenes into simulation, builds simulation-ready 3D backgrounds and assets, and supports scalable benchmarking, synthetic data generation, and reward learning.
From Real Scenes to Simulation-Ready Worlds
A central part of this project is reconstructing real-world scenes inside simulation. Starting from visual input, we explore ways to recover complete 3D backgrounds using video generation, video-to-world reconstruction, and image-to-world models. The reconstructed background is then aligned with the simulated robot and camera setup through pose and scale optimization, producing an explorable 3D environment that can support robot-centered evaluation and learning.