Srinath Ravi, Fernando De la Torre
3D Gaussian Splatting (3DGS) delivers impressive real-time scene reconstruction, yet it often struggles with “floaters” and blurred textures when viewed from angles outside the original training data. RefineGS solves this by introducing a post-hoc, image-space correction module that cleans up corrupted renders without altering the underlying 3D representation. By leveraging the CroCo v2 backbone for geometric consistency and a Conditional Flow Matching (CFM) framework for efficient, deterministic refinement, RefineGS restores photometric accuracy to unseen views. Our framework provides a principled, high-performance solution for eliminating artifacts and ensuring structural integrity in modern splat-based pipelines.
RefineGS
Instead of the computationally expensive task of re-optimizing Gaussian primitives, we propose a modular, image-space solution. RefineGS treats initial renders as “noisy” starting points and refines them using a learned transformer that understands 3D relationships.
Contributions:
- Cross-View Completion: By utilizing a weight-sharing encoder, our architecture establishes robust geometric links between a corrupted render and a high-quality reference view. This allows the model to fix missing details and resolve structural ambiguities.
- Deterministic Flow: To meet the low-latency requirements of VR and interactive tools, we move away from slow, stochastic diffusion. Our use of Conditional Flow Matching (CFM) enables efficient refinement via numerical ODE integration.
- Large-Scale Training: We developed a specialized pipeline using the DL3DV 10K dataset to produce 1 million paired views. This provides the explicit supervision needed for the model to recognize and correct 3DGS-specific noise patterns.
Past Methods & Related Work
Representation-Level Optimization
Many existing solutions attempt to fix these issues by regularizing the 3D model during training.
- Geometric Constraints: Methods like SplatFormer and VA-GS use transformers or visibility-aware regularizers to encourage geometric consistency.
- Surface Priors: Tools such as GS-Pull and PGSR pull Gaussians toward a coherent topology to improve surface fidelity. While effective, these approaches suffer from unpredictable latency—as the number of primitives scales with scene complexity—and they require deep access to the optimization loop, which is not always possible in industrial pipelines using pre-trained assets.
Image-Space and Generative Refinement
An alternative strategy treats the rendered output as a corrupted signal.
- Diffusion Models: Recent generative models like Difix3D+ and RI3D produce high-quality results but are extremely slow (often 0.5 FPS) due to the iterative nature of latent diffusion.
- Feed-Forward Networks: Models like 3DGS-Enhancer offer faster single-pass networks but often fail to resolve the specific “tearing” and flickering artifacts unique to 3DGS. Most of these models also lack the multi-view geometric conditioning needed for true consistency.
Where RefineGS Fits In
RefineGS combines the speed of feed-forward networks with the geometric intelligence of 3D-centric methods. By leveraging the CroCo v2 backbone for cross-view completion and Conditional Flow Matching (CFM) for deterministic refinement, we avoid the high costs of diffusion and the unpredictability of primitive-level optimization. Trained on over 1 million paired views from the DL3DV-10K dataset, RefineGS is purpose-built to recognize and correct 3DGS-specific noise patterns in real time.
Method
Problem Formulation
3D Gaussian Splatting (3DGS) often fails when training trajectories are sparse, as the optimizer lacks sufficient geometric constraints. This leads to structured artifacts—specifically “floaters,” surface tearing, and incorrect specularities in unobserved regions.
Our approach learns a deterministic correction function. Instead of generating a scene from scratch, we map a noisy out-of-distribution (OOD) render and a nearby high-quality reference view into a corrected, clean image. By using the reference view as a geometric anchor, the model resolves visual ambiguities through cross-view consistency.
Dataset Curation
We developed a paired training corpus using the DL3DV-10K dataset through a two-stage process:
Proof of Concept (POC): Initially, we validated our denoising logic by manually perturbing the attributes (position, opacity, and scale) of a subset of Gaussians in a controlled scene. This allowed us to generate synthetic but perfectly aligned noisy/ground-truth pairs for early testing.
Clustering-Based Failures: For the final pipeline, we induce realistic failures by performing k-means clustering on camera poses. By concentrating training views in specific clusters, we intentionally leave gaps in the scene coverage.
Multi-View Feature Encoding
To identify and fix artifacts, we use a weight-sharing encoder based on CroCo v2. Both the target noisy image and the reference image are processed by the same ViT-Base backbone.
- Unified Feature Space: Processing both images through the same weights ensures their latent representations are aligned, making it easier for the model to find correspondences.
- Rectangular RoPE: Standard positional embeddings often struggle with high-definition rectangular aspect ratios (544 × 960). We implement 2D Rectangular Rotary Positional Embeddings to maintain precise spatial awareness across the entire wide-frame render.
Just-in-Time (JiT) Refinement Head
The encoded features pass into a JiT Refinement Head for final processing:
Temporal Conditioning: To support the Flow Matching process, the head is conditioned on a time-step $t$, which modulates how intensely the model refines the image at different stages of the integration.
Cross-View Attention: Target patches “attend” to reference patches. If the target view shows a “floater” that isn’t present in the reference view’s geometry, the attention mechanism identifies it as noise.
Conditional Flow Matching (CFM)
We treat the refinement as a Flow Matching problem rather than a stochastic diffusion process.
Velocity Prediction: The model is trained to predict the “velocity” vector field required to transform the noisy pixels into clean ones. This deterministic approach is more stable and faster to train than standard diffusion.
Linear Probability Path: We define a direct path from the noisy render to the ground truth.
Inference & Optimization
At inference time, we recover the clean image by integrating the predicted velocity field using Heun’s Method (a second-order solver).
Performance Optimization: To keep inference efficient, we implement feature caching. We compute the expensive backbone features for the reference view only once. During the iterative refinement steps, we only update the lightweight tokens for the target view, significantly reducing the computational load.
Results
| Method | PSNR | SSIM | Inference Speed (FPS) |
| Noisy Input (Perturbed 3DGS) | 21.94 | 0.717 | – |
| Difix3D+ (SOTA) | 20.65 | 0.548 | 0.5 |
| Ours | 23.56 | 0.724 | 20.45 |
Qualitative analysis confirms that RefineGS effectively
“borrows” sharp textures and geometric structure from the
reference view to resolve target artifacts. The model is particularly successful at removing “foggy” floaters and correcting edge-tearing artifacts that typically result from the
3DGS optimizer’s failure to constrain geometry in low-coverage regions. By leveraging pre-trained geometric priors from the CroCo backbone, the model maintains high-frequency details that are often lost in generic feed-forward denoising approaches.
Resources