Overview
Our method aims to construct simulation-ready 3D environments from real-world visual observations. The goal is to make reconstructed scenes usable inside robotic simulation, where they can support camera rendering, benchmarking, synthetic data generation, reward learning, and future Vision–Language–Action model training.
The pipeline contains three main stages: first, reconstructing a full 3D scene background from visual input; second, aligning the reconstructed background with the simulated robot and camera setup; and third, integrating the aligned scene into simulation as an explorable environment for robot-centered perception and learning.
The overall pipeline consists of the following stages:
- Real-world visual input
We begin with visual observations of a real environment, such as an image or video captured from a camera. - 3D scene reconstruction
The visual input is converted into a 3D scene representation using image-to-world, video-to-world, or video-to-Gaussian reconstruction methods. - Background alignment
The reconstructed background is refined in pose and scale so that it matches the simulated camera and robot coordinate system. - Simulation integration
The aligned 3D scene is imported into simulation as a renderable and explorable environment. - Learning applications
The resulting environment can support benchmarking, synthetic data generation, reward learning, and future VLA policy training.
1. Real-World Visual Input
The pipeline begins with visual observations of a real-world manipulation scene. These observations provide the appearance and spatial context of the environment that we want to reconstruct inside simulation.
In the current setting, visual input may come from a fixed camera or a mobile camera. The input scene can include the workspace, background structure, and task-relevant objects. This visual information serves as the starting point for reconstructing a simulation-ready environment.
2. 3D Scene Reconstruction
To bring real-world environments into simulation, we first recover a full 3D background from visual input. We explore two main directions:
- Video generation followed by video-to-world reconstruction
- Off-the-shelf image-to-world reconstruction models
The reconstructed 3D background provides a visually rich representation of the environment. However, the raw reconstruction is usually not directly usable for robotic simulation because its coordinate frame, camera alignment, and physical scale may not match the simulator.
3. Background Alignment
After reconstruction, the 3D background must be aligned with the simulated robot and camera setup. This step is necessary because the reconstructed scene may have inaccurate rotation, translation, or scale.
Even if the scene looks visually plausible, it may still be geometrically inconsistent with the simulator. For robotic manipulation, this mismatch is important: the operating surface, object locations, and camera views must be aligned with the simulated robot workspace.
The alignment stage therefore addresses two key problems:
- Pose alignment: correcting rotation and translation
- Scale alignment: correcting the physical size of the reconstructed background
4. Pose Optimization
The first alignment problem is pose. A reconstructed background may be shifted or rotated relative to the simulated camera and robot coordinate system. To correct this, we optimize the background pose by comparing rendered simulation views against reference visual observations.
The pose objective combines low-level visual consistency with feature-level similarity. In simple terms, the optimization adjusts the background placement until the rendered view better matches the target scene.
A technical version of the objective can be written as:
where the RGB term encourages pixel-level consistency and the DINOv2 feature term encourages semantic and perceptual alignment.
5. Scale Optimization
The second alignment problem is scale. A reconstructed scene may have a plausible appearance but an incorrect physical size. This is especially problematic for robotic manipulation because the robot must interact with surfaces and objects at the correct metric scale.
To address this, we refine the scene scale by comparing geometry from a metric-depth mesh with the collision mesh of the reconstructed background. Instead of matching the entire scene indiscriminately, the optimization focuses on selected points inside the camera frustum, allowing the scale estimate to better reflect the visible workspace and operating surface.
A technical version of the objective can be written as:
where denotes selected vertices from the metric-depth mesh, and denotes selected vertices from the collision mesh.
6. Simulation Integration
Once pose and scale are refined, the reconstructed background can be integrated into simulation as a renderable and explorable environment. This allows the simulator to produce views from different camera configurations, including the main camera and wrist-camera viewpoints.
This step is important because the final environment should not only look realistic, but also support robot-centered perception and evaluation. The aligned scene provides a foundation for future tasks such as policy benchmarking, synthetic data generation, and reward-guided learning.
7. Learning-Oriented Applications
The final goal of this pipeline is to support scalable robot learning. Once realistic 3D environments are available in simulation, they can be used for multiple downstream applications:
- Benchmarking: evaluating robotic policies in realistic reconstructed environments
- Synthetic data generation: creating diverse training data with controllable scene and task variation
- Reward learning: training or applying learned reward models in simulated environments
- VLA training: improving robustness and generalization through scalable simulation data
This makes the reconstructed scene more than a visualization artifact. It becomes part of a broader learning pipeline for scalable robotic manipulation.
Method Summary
In summary, our method reconstructs real-world scenes into simulation-ready 3D environments, aligns them through pose and scale optimization, and integrates them into robotic simulation. This pipeline provides a foundation for realistic benchmarking, scalable synthetic data generation, reward learning, and future Vision–Language–Action model training.