Our goal is to build an efficient system for streaming free-viewpoint video: a live reconstruction that lets viewers move through a dynamic real-world scene from any perspective. While traditional 3D reconstruction captures a scene at a single moment in time, dynamic reconstruction extends this idea to 4D by updating the scene frame by frame. In a streaming setting, where multiple cameras capture an event as it unfolds, this could create much richer live experiences! Sports fans could move around and see key plays from the perspective of the players, remote concertgoers could step into a performance through AR/VR, and telepresence systems could make a person appear as a lifelike 3D projection in the same room. 3D Gaussian Splatting has become a strong foundation for this kind of work because it can render high-quality scenes much faster than earlier neural rendering methods such as NeRFs.
Building streaming free-viewpoint video is difficult because the reconstruction must be updated quickly, compactly, and accurately as the scene changes over time. After the first 3D Gaussian scene is trained, each new frame requires the system to adjust the Gaussians so the reconstruction remains aligned with the real world. This creates three core challenges:
- Training speed: For a true streaming experience, each frame must be processed fast enough to keep up with live video. However, even state-of-the-art methods can still take seconds to update a single frame, which is far from the pace needed for real-time 30 FPS streaming.
- Bandwidth and storage: A high-quality 4D scene contains a large amount of information, including geometry, appearance, and motion over time. If each frame requires too much data to store or transmit, the system becomes impractical for real-time streaming.
- Visual fidelity: Speed and compression often come at the cost of quality. Heavy quantization, minimal training, or overly simplified motion updates can introduce artifacts, blurry details, or unrealistic geometry, which break the sense of immersion.
Beyond these core tradeoffs, dynamic scenes introduce additional technical challenges: large motions may require major updates to the Gaussian field, new or disappearing objects may require adding and pruning Gaussians, and difficult visual effects such as reflections, occlusions, sparse camera views, or low-texture regions can make accurate reconstruction even harder.