Splat and Replace: 3D Reconstruction with Repetitive Elements

TL;DR

This paper introduces a method that leverages repetitive elements in 3D scenes to enhance the quality of novel view synthesis by sharing information among instances in 3D Gaussian Splatting reconstructions.

Contribution

It proposes a novel segmentation and registration approach to exploit scene repetitions, improving geometry and appearance modeling in 3D reconstruction.

Findings

Significant improvement in novel view synthesis quality.

Effective segmentation and registration of repeated scene elements.

Enhanced reconstruction of occluded and poorly covered scene parts.

Abstract

We leverage repetitive elements in 3D scenes to improve novel view synthesis. Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have greatly improved novel view synthesis but renderings of unseen and occluded parts remain low-quality if the training views are not exhaustive enough. Our key observation is that our environment is often full of repetitive elements. We propose to leverage those repetitions to improve the reconstruction of low-quality parts of the scene due to poor coverage and occlusions. We propose a method that segments each repeated instance in a 3DGS reconstruction, registers them together, and allows information to be shared among instances. Our method improves the geometry while also accounting for appearance variations across instances. We demonstrate our method on a variety of synthetic and real scenes with typical repetitive elements, leading to a substantial improvement in the quality of novel view synthesis.