R3GS: Gaussian Splatting for Robust Reconstruction and Relocalization in Unconstrained Image Collections

TL;DR

R3GS introduces a robust 3D reconstruction and relocalization framework for unconstrained image collections, utilizing hybrid representations, transient object mitigation, sky handling, and lighting-invariant pose estimation to improve fidelity and efficiency.

Contribution

The paper presents a novel hybrid representation, transient object removal, sky handling with depth priors, and a lighting-robust relocalization method, advancing 3D reconstruction and relocalization in uncontrolled datasets.

Findings

Achieves state-of-the-art performance on in-the-wild datasets.

Enhances rendering fidelity and efficiency.

Reduces storage requirements.

Abstract

We propose R3GS, a robust reconstruction and relocalization framework tailored for unconstrained datasets. Our method uses a hybrid representation during training. Each anchor combines a global feature from a convolutional neural network (CNN) with a local feature encoded by the multiresolution hash grids [2]. Subsequently, several shallow multi-layer perceptrons (MLPs) predict the attributes of each Gaussians, including color, opacity, and covariance. To mitigate the adverse effects of transient objects on the reconstruction process, we ffne-tune a lightweight human detection network. Once ffne-tuned, this network generates a visibility map that efffciently generalizes to other transient objects (such as posters, banners, and cars) with minimal need for further adaptation. Additionally, to address the challenges posed by sky regions in outdoor scenes, we propose an effective sky-handling technique that incorporates a depth prior as a constraint. This allows the inffnitely distant sky to be represented on the surface of a large-radius sky sphere, signiffcantly reducing ffoaters caused by errors in sky reconstruction. Furthermore, we introduce a novel relocalization method that remains robust to changes in lighting conditions while estimating the camera pose of a given image within the reconstructed 3DGS scene. As a result, R3GS significantly enhances rendering ffdelity, improves both training and rendering efffciency, and reduces storage requirements. Our method achieves state-of-the-art performance compared to baseline methods on in-the-wild datasets. The code will be made open-source following the acceptance of the paper.