Augment training images using diffusion models
While 360° video capture provides dense multi-view imagery as the photographer moves through the scene, this walking-based capture results in input images being constrained to a nearly constant elevation. Although this is sufficient for training views along the capture path, geometric consistency, particularly for tall structures and distant objects, degrades as viewpoints move farther away from the trajectory.
One way to address this limitation is to synthesize additional training views at a fixed elevated height (e.g., +1.5–2 m) offset from the original path, and refine these views using a diffusion-based model such as Difix3D+. These synthesized views can be incorporated into training to increase dataset coverage, especially in regions that are sparsely observed in the original capture.
The same diffusion-based approach can also be employed at inference time as a learned rasterizer to suppress Gaussian Splatting artifacts in real-time novel view synthesis, further improving visual fidelity.
Here are some examples of using Difix3D+ to improve visual fidelity and details in the novel view renders.
Semantic Filtering
We can further improve reconstruction quality by removing dynamic distractors through semantic segmentation. Using equirectangular instance segmentation—or, alternatively, performing segmentation on cubemap projections—we generate masks for pedestrians, vehicles, and animals, and exclude these regions during Gaussian Splatting scene reconstruction.
The examples below illustrate how such dynamic distractors introduce artifacts and degrade geometric consistency in GS reconstructions when left unfiltered.
Adaptive Sampling
To optimize efficiency during training and evaluation, we could implement adaptive subsampling strategies guided by the confidence maps. Instead of using all frames uniformly, the system will prioritize high-confidence, spatially diverse views, reducing redundancy and computational overhead without sacrificing reconstruction quality.