Distractor-free Generalizable 3D Gaussian Splatting

TL;DR

DGGS introduces a novel framework for 3D Gaussian Splatting that effectively handles distractor data, improving stability, generalization, and inference quality in unseen scenes through innovative mask prediction and distractor removal techniques.

Contribution

It proposes a scene-agnostic, reference-based mask prediction and a two-stage inference framework to eliminate distractor effects in 3D Gaussian Splatting, enhancing generalization and stability.

Findings

Outperforms existing methods in distractor scene reconstruction.

Achieves superior mask prediction accuracy compared to scene-specific training.

Effectively removes distractor artifacts and holes during inference.

Abstract

We present DGGS, a novel framework that addresses the previously unexplored challenge: $\textbf{Distractor-free Generalizable 3D Gaussian Splatting}$ (3DGS). It mitigates 3D inconsistency and training instability caused by distractor data in the cross-scenes generalizable train setting while enabling feedforward inference for 3DGS and distractor masks from references in the unseen scenes. To achieve these objectives, DGGS proposes a scene-agnostic reference-based mask prediction and refinement module during the training phase, effectively eliminating the impact of distractor on training stability. Moreover, we combat distractor-induced artifacts and holes at inference time through a novel two-stage inference framework for references scoring and re-selection, complemented by a distractor pruning mechanism that further removes residual distractor 3DGS-primitive influences. Extensive feedforward experiments on the real and our synthetic data show DGGS's reconstruction capability when dealing with novel distractor scenes. Moreover, our generalizable mask prediction even achieves an accuracy superior to existing scene-specific training methods. Homepage is https://github.com/bbbbby-99/DGGS.