Key Idea:
Our key idea is to leverage optical flow as a motion prior for a 4D Gaussian Splatting system. Since optical flow estimates how pixels move between consecutive video frames, it gives us a useful signal about scene motion without requiring any additional capture hardware. In our setting, we compute optical flow across the 2D frames from a multi-view camera setup, lift that motion into 3D through triangulation, and use it to guide how the Gaussian field should update over time. This should reduce the amount of training needed for each new frame, since the system is no longer starting from scratch when estimating motion. We also explore complementary ideas such as flow-based regularization, rigidity losses for more coherent object and human motion, and dynamic/static separation so that computation is focused on the parts of the scene that actually change. After improving update speed, we aim to further reduce storage and bandwidth costs by organizing the 3D scene with efficient structures such as hash grids and octrees, allowing the system to better exploit spatial and temporal redundancy.
We propose, FlowGS which is built on top of 3DGStream and introduces optical flow as a geometric prior to accelerate per-frame Gaussian deformation. Our pipeline has four interconnected stages: (1) optical flow estimation and trajectory building, (2) 2D→3D flow lifting via a modified rasterizer and multi-view triangulation, (3) a flow-conditioned Neural Transformation Cache (NTC), and (4) a suite of regularizers that improve stability and generalization across frames.
1. Optical Flow Estimation
We evaluate three optical flow model: RAFT, UFM, and MegaFlow (suited for large motions and fine boundaries) to produce dense per-pixel motion fields between consecutive frames.
Two dataset-specific challenges required targeted solutions. For the RunningBlack sequence, textureless black regions produce noisy, unconstrained optical flow with no matchable features. We address this by injecting a static Perlin noise texture over these areas, giving the flow estimator matchable gradients while preserving zero true motion signal. For the Basketball Stadium sequence, fast and complex motion causes frame-to-frame drift when using pairwise optical flow naively. We instead apply Sobel edge sampling to select sparse high-motion pixels, then run CoTracker to build temporally coherent trajectories across the full clip, attributing them back to Gaussians and eliminating the drift that comes from pairwise estimation.
2. Modified Rasterizer and 2D→3D Triangulation
Standard 3DGS rasterization discards per-pixel Gaussian attribution after alpha compositing. We modify the CUDA rasterizer to output the top-K Gaussians per pixel along with their α·T (alpha-transmittance) weights. This is the key enabler: it lets us trace a 2D flow vector at any pixel back to the specific 3D Gaussians responsible for rendering it.
For triangulation, each Gaussian’s depth Zc linearizes the projection, turning a 3D velocity into two flow constraints per camera view. Since Zc couples the depth-axis velocity vz to the observed 2D flow, combining multiple camera views makes the system solvable. For each pixel we sample RAFT flow, distribute it to the top-K Gaussians weighted by α·T, and solve per-Gaussian 3×3 linear systems with Tikhonov regularization. Occluded Gaussians are filtered using depth maps from DepthAnything.
We further improve the solve quality through camera selection: each camera is scored by its angular orthogonality to the estimated motion direction. Cameras whose viewing direction is perpendicular to the motion see displacement cleanly in-plane and produce well-conditioned constraints, while cameras nearly parallel to the motion produce degenerate ones. We keep the top-K most orthogonal cameras and rerun the solve on this reduced system to obtain depth-aligned 3D velocity vectors per Gaussian.
3. Flow-Conditioned Neural Transformation Cache
3DGStream’s NTC uses an Instant-NGP-style spatial hash encoding to predict per-Gaussian 3D translation (d_xyz) at each new frame. We extend this by concatenating the triangulated 3D flow vector directly with the NTC’s spatial hash features before the MLP head, giving the network a strong per-frame motion prior that accelerates convergence without per-Gaussian fine-tuning or data-dependent pretraining. The architecture is shown below.
This directly addresses limitations of prior work: IGS displaces Gaussians feedforward from 2D flow features but errors cascade without densification. TrackerSplat triangulates flow geometrically but still requires ~1k slow fine-tuning iterations and depends heavily on dense point-tracking quality. Our approach provides the geometric grounding of triangulation while keeping the NTC’s fast per-frame optimization loop intact.
4. Loss Functions and Regularizers
We introduce two complementary supervision signals on the predicted Gaussian deformations. Flow3DLoss (L3D) directly penalizes the predicted per-Gaussian 3D translation d_xyz for deviating from the triangulated 3D flow vectors — fast to compute but inherits noise from the triangulation solve. Flow2DReprojectedLoss (L2D) takes a cleaner approach: it projects each Gaussian’s deformed position into each camera view and penalizes the predicted 2D pixel displacement against the raw optical flow, bypassing triangulation noise entirely and providing direct photometric grounding.
We also adopt the local rigidity loss from Luiten et al. (Dynamic 3D Gaussians). For each Gaussian i, its k-nearest neighbors in the canonical frame should move as if rigidly attached to i’s local coordinate frame — deviations are penalized with a Gaussian falloff on initial inter-particle distance, enforcing local rigidity while permitting global non-rigidity. Companion losses on rotation similarity and long-term isometry prevent trajectory drift over time. This regularizer fills in low-texture regions by propagating reliable velocity from textured neighbors, adds a spatial prior on the fragile depth-axis velocity component vz, and couples neighboring NTC-predicted trajectories to smooth out per-particle noise.
To restrict deformation modeling to only regions that actually move, we evaluate three dynamic partitioning strategies. Viewspace Gradient Difference (VGD) constructs a vanilla 3DGS scene at frame t−1 and compares its viewspace gradients against those at frame t — a high gradient difference signals the scene has changed, identifying dynamic Gaussians. GMM-based Motion Partitioning renders the canonical scene at t=0, computes an image difference with a future frame, and clusters the resulting gradients; the bimodal distribution cleanly separates high-gradient dynamic Gaussians from static ones. Optical Flow Partitioning aggregates flow magnitudes per Gaussian across all camera views and applies a threshold — the most direct approach for a flow-guided pipeline.
Experiments
We evaluate our method on two challenging sequences: RunningBlack and Basketball Stadium. As a representative ablation on one sample, the 3DGStream baseline achieves 23.07 dB PSNR. Adding flow conditioning with the direct 3D loss (L3D) raises this to 31.05 dB, a meaningful improvement, though the solve’s triangulation noise limits the ceiling. Switching to the reprojection-based loss (L2D) jumps substantially further to 38.95 dB, confirming that bypassing triangulation noise and supervising directly in image space provides much cleaner gradients. Adding the local rigidity regularizer on top of L2D brings the result to 37.65 dB vs. 35.01 dB without it, demonstrating that rigidity meaningfully stabilizes trajectories in low-texture regions and improves overall reconstruction quality.
On the compression side, we benchmark two approaches for the static scene representation. Octree-GS achieves 28.43 MB at 24.83 dB PSNR. HAC++ dramatically reduces this to just 4.02 MB at 24.39 dB, a 100×+ storage reduction over vanilla 3DGS with only a marginal quality drop, validating the path toward bandwidth-efficient 4D streaming.
Future Work
The central remaining challenge is MLP obedience to flow. The NTC must simultaneously learn how to interpret the 3D flow signal and how much to trust it, since triangulation quality varies with camera geometry and texture density. This introduces an unavoidable warmup cost per frame. We are exploring Eulerian and Lagrangian solvers that can be analytically initialized from triangulated flow and iteratively refined, which would remove warmup overhead entirely and make the deformation field directly interpretable.
A related structural issue is that drifting Gaussians render old hash weights meaningless, forcing full NTC retraining every frame. Decoupling the spatial hash encoding from the motion prediction head a “delineated NTC” would allow the flow and structure representations to be cheaply reused across frames without full reinitialization, significantly reducing per-frame training time.
We also plan to explore explicit structure via convex hulls: modeling canonical-frame geometry through per-object convex hulls could provide stronger structural priors in low-texture regions where flow is unreliable and rigidity regularization alone is insufficient. For the broader pipeline, end-to-end flow and deformation training jointly optimizing the flow estimator and NTC would let the network learn motion representations optimized directly for 3D reconstruction quality rather than 2D pixel accuracy, potentially eliminating the noise-obedience tradeoff entirely.
On the streaming side, we plan to extend the Octree data structure as the underlying spatial representation, which builds in level-of-detail natively: distant or slow-moving regions can use coarser resolution to save memory and bandwidth, while busy regions preserve full quality. Finally, packaging dynamic Gaussian residuals into H.265/AV1-compatible representations would enable 4D streaming over existing video infrastructure without custom pipelines, making the system deployable at scale for applications in sports, telepresence, and live performance streaming.
We plan to submit some of our findings on lifting 2D Optical Flow for faster reconstructions with HAC++ enabled compressions to Octree’s in a streaming setup to 3DV