Robust 3D Gaussian Splatting for Novel View Synthesis in Presence of Distractors

TL;DR

This paper introduces a self-supervised method that uses residual analysis and pretrained segmentation to identify and exclude distractors in 3D Gaussian Splatting, significantly enhancing view synthesis quality in scenes with dynamic objects.

Contribution

It presents a novel approach combining residual-based detection and segmentation masks to robustly ignore distractors during 3D Gaussian optimization.

Findings

Improves PSNR by 1.86dB on distractor scenes.

Effectively excludes distractors, reducing artifacts.

Enhances robustness of 3D Gaussian Splatting in dynamic environments.

Abstract

3D Gaussian Splatting has shown impressive novel view synthesis results; nonetheless, it is vulnerable to dynamic objects polluting the input data of an otherwise static scene, so called distractors. Distractors have severe impact on the rendering quality as they get represented as view-dependent effects or result in floating artifacts. Our goal is to identify and ignore such distractors during the 3D Gaussian optimization to obtain a clean reconstruction. To this end, we take a self-supervised approach that looks at the image residuals during the optimization to determine areas that have likely been falsified by a distractor. In addition, we leverage a pretrained segmentation network to provide object awareness, enabling more accurate exclusion of distractors. This way, we obtain segmentation masks of distractors to effectively ignore them in the loss formulation. We demonstrate that our approach is robust to various distractors and strongly improves rendering quality on distractor-polluted scenes, improving PSNR by 1.86dB compared to 3D Gaussian Splatting.