Recent work in streaming and dynamic 3D Gaussian Splatting has shaped our project from several angles: how to update a scene quickly over time, how to avoid wasting computation on static regions, how to structure Gaussians more efficiently, and how to use motion cues from optical flow rather than relying only on reconstruction loss.
3DGStream showed that free-viewpoint video can be reconstructed in a streaming, frame-by-frame manner instead of requiring a full video sequence to be trained offline. Its key insight is to avoid directly optimizing all Gaussians at every timestep; instead, it uses a compact Neural Transformation Cache (“NTC”) to predict Gaussian translations and rotations, which reduces per-frame training and storage while maintaining real-time rendering. This is the closest baseline to our setting because it directly targets streamable dynamic scenes
QUEEN builds on the idea of frame-to-frame residual updates, but focuses more heavily on compact encoding. Rather than storing full Gaussian updates, it learns and compresses attribute residuals between consecutive frames using a quantization-sparsity framework. Its most relevant idea for us is using viewspace gradient differences to identify which Gaussians actually belong to dynamic regions, so training and streaming can focus on the parts of the scene that actually change.
Scaffold-GS addresses a different but important issue: raw 3D Gaussian clouds can be redundant and poorly organized. It introduces a sparse grid of anchor points, where each anchor generates local neural Gaussians and predicts their attributes based on viewing direction and distance. This motivates the idea that a more structured Gaussian representation can reduce redundancy while still preserving high-quality rendering
Octree-GS extends this structure further by organizing anchors into a hierarchy of levels of detail. Instead of rendering every Gaussian equally, it dynamically selects the appropriate level based on the view, so distant views can use coarser detail while closer views can access finer structure. Although this work is only for static and not temporal scenes, it is still relevant to our long-term goal of reducing storage and bandwidth by exploiting spatial redundancy in the Gaussian field.
Motion-Aware 3DGS is particularly important for our optical-flow direction. It argues that dynamic 3DGS methods often overlook the motion information already present in 2D video, then uses optical flow to connect pixel-level motion with 3D Gaussian movement. Its use of flow-based motion correspondence, uncertainty-aware flow losses, and dynamic-aware optimization supports our core intuition that optical flow can act as a useful motion prior for faster and more coherent Gaussian updates
Together, these works show strong progress toward streamable dynamic Gaussian reconstruction, but important limitations remain. Per-frame updates can still be too slow for true real-time streaming, compact representations can trade off against visual quality, and many methods still rely heavily on optimization to discover motion rather than using motion cues directly. Our project aims to address this gap by using optical flow as an explicit motion signal, separating dynamic and static regions, and eventually combining motion-aware updates with efficient scene structures such as hash grids and octrees.
References:
- Sun, Jiakai, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. “3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos.” CVPR, 2024.
- Girish, Sharath, Tianye Li, Amrita Mazumdar, Abhinav Shrivastava, David Luebke, and Shalini De Mello. “QUEEN: QUantized Efficient ENcoding of Dynamic Gaussians for Streaming Free-viewpoint Videos.” NeurIPS, 2024.
- Lu, Tao, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. “Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering.” CVPR, 2024.
- Ren, Kerui, Lihan Jiang, Tao Lu, Mulin Yu, Linning Xu, Zhangkai Ni, and Bo Dai. “Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians.” arXiv preprint arXiv:2403.17898, 2024.
- Guo, Zhiyang, Wengang Zhou, Li Li, Min Wang, and Houqiang Li. “Motion-Aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction.” arXiv preprint arXiv:2403.11447, 2024.