GaussFusion: Improving 3D Reconstruction in the Wild with A Geometry-Informed Video Generator

TL;DR

GaussFusion enhances 3D Gaussian splatting reconstructions in the wild by using a geometry-informed video generator to reduce artifacts and improve visual quality, achieving real-time performance.

Contribution

It introduces a geometry-informed video generator that refines 3D reconstructions and a robust artifact synthesis pipeline for improved 3D scene rendering.

Findings

State-of-the-art results on novel-view synthesis benchmarks.

Real-time variant runs at 15 FPS with comparable quality.

Effectively mitigates artifacts like floaters, flickering, and blur.

Abstract

We present GaussFusion, a novel approach for improving 3D Gaussian splatting (3DGS) reconstructions in the wild through geometry-informed video generation. GaussFusion mitigates common 3DGS artifacts, including floaters, flickering, and blur caused by camera pose errors, incomplete coverage, and noisy geometry initialization. Unlike prior RGB-based approaches limited to a single reconstruction pipeline, our method introduces a geometry-informed video-to-video generator that refines 3DGS renderings across both optimization-based and feed-forward methods. Given an existing reconstruction, we render a Gaussian primitive video buffer encoding depth, normals, opacity, and covariance, which the generator refines to produce temporally coherent, artifact-free frames. We further introduce an artifact synthesis pipeline that simulates diverse degradation patterns, ensuring robustness and generalization. GaussFusion achieves state-of-the-art performance on novel-view synthesis benchmarks, and an efficient variant runs in real time at 15 FPS while maintaining similar performance, enabling interactive 3D applications.