Synthetic Data for Scalable Robot Learning
Recent work has shown that simulation data can play an important role in training generalist robotic policies. In particular, InternData-A1 demonstrates that high-fidelity synthetic data can provide strong pretraining signals for Vision–Language–Action models. The work suggests that simulation-only data and real-world data offer complementary benefits, and that a practical strategy is to first pretrain on large-scale simulation data before adapting to real-world settings.
This direction is closely related to our project because it highlights the importance of building realistic, diverse, and scalable simulation environments. While synthetic data generation requires task variation, robot control, and trajectory generation, it also depends on the quality and realism of the simulated scenes. Our project contributes to this foundation by exploring how real-world scenes can be reconstructed and aligned into simulation-ready 3D environments.
Real-to-Sim Translation and Scene Reconstruction
Another important line of work studies how real-world scenes and robot setups can be translated into simulation. RobotArena ∞ demonstrates a scalable real-to-sim pipeline for robot benchmarking. Its pipeline includes robot-camera calibration, scene reconstruction, asset pose alignment, and system identification, showing how digital twins can support efficient evaluation of real-world scenarios in simulation.
This direction is the most directly connected to our project. We similarly aim to bring real-world environments into simulation, but our current focus is on reconstructing and aligning full 3D scene backgrounds so that they become explorable and simulation-ready. In particular, we study the challenges of background pose alignment and scale matching, which are necessary for making reconstructed scenes useful for robot-centered rendering and interaction.
Reward Learning for Robotic Manipulation
Reward design is another major challenge in robot learning. Handcrafted rewards are often task-specific and difficult to scale, while sparse binary rewards may not provide enough learning signal for long-horizon manipulation. RoboReward addresses this challenge by learning vision-language reward models that estimate task progress from visual observations and language instructions. Its results suggest that progress-based reward signals can improve reinforcement learning efficiency and policy performance.
This line of work is important for our broader project direction because realistic simulation environments become more powerful when paired with scalable learning signals. By combining reconstructed simulation environments with reward learning, future systems can generate interaction data, evaluate task progress, and refine policies more efficiently.
Broader Research Context
Together, these works point toward a broader research direction: scaling robot learning by combining realistic simulation environments, synthetic data generation, and learned reward signals. Instead of relying only on expensive real-world demonstrations, robotic systems can benefit from simulation pipelines that generate diverse data, evaluate policies safely, and provide richer feedback during training.
Our project sits at the intersection of these ideas. We focus on reconstructing real-world scenes into simulation-ready 3D environments, aligning them with camera and robot setups, and preparing them for future use in benchmarking, synthetic data generation, reward learning, and VLA training.
References
[1] Y. Tian, Y. Yang, Y. Xie, Z. Cai, X. Shi, N. Gao, H. Liu, X. Jiang, Z. Qiu, F. Yuan, Y. Li, P. Wang, J. Cai, J. Zeng, H. Dong, and J. Pang, “InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policy,” arXiv:2511.16651, 2025.
[2] Y. Jangir, Y. Zhang, P.-C. Lo, K. Yamazaki, C. Zhang, K.-H. Tu, T.-W. Ke, L. Ke, Y. Bisk, and K. Fragkiadaki, “RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translation,” arXiv:2510.23571, 2025.
[3] T. Lee, A. Wagenmaker, K. Pertsch, P. Liang, S. Levine, and C. Finn, “RoboReward: General-Purpose Vision-Language Reward Models for Robotics,” arXiv:2601.00675, 2026.