Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections

TL;DR

Wild-GS is a novel 3D Gaussian Splatting-based method for real-time, high-quality novel view synthesis from unconstrained photo collections, effectively handling transient occlusions and appearance variations.

Contribution

Wild-GS adapts 3D Gaussian Splatting for unconstrained photos, explicitly aligning pixel features to local Gaussians, enabling faster training and superior rendering quality.

Findings

Achieves state-of-the-art rendering performance.

Offers the highest efficiency in training and inference.

Effectively handles transient occlusions and appearance variations.

Abstract

Photographs captured in unstructured tourist environments frequently exhibit variable appearances and transient occlusions, challenging accurate scene reconstruction and inducing artifacts in novel view synthesis. Although prior approaches have integrated the Neural Radiance Field (NeRF) with additional learnable modules to handle the dynamic appearances and eliminate transient objects, their extensive training demands and slow rendering speeds limit practical deployments. Recently, 3D Gaussian Splatting (3DGS) has emerged as a promising alternative to NeRF, offering superior training and inference efficiency along with better rendering quality. This paper presents Wild-GS, an innovative adaptation of 3DGS optimized for unconstrained photo collections while preserving its efficiency benefits. Wild-GS determines the appearance of each 3D Gaussian by their inherent material attributes, global illumination and camera properties per image, and point-level local variance of reflectance. Unlike previous methods that model reference features in image space, Wild-GS explicitly aligns the pixel appearance features to the corresponding local Gaussians by sampling the triplane extracted from the reference image. This novel design effectively transfers the high-frequency detailed appearance of the reference view to 3D space and significantly expedites the training process. Furthermore, 2D visibility maps and depth regularization are leveraged to mitigate the transient effects and constrain the geometry, respectively. Extensive experiments demonstrate that Wild-GS achieves state-of-the-art rendering performance and the highest efficiency in both training and inference among all the existing techniques.