MuGS: Multi-Baseline Generalizable Gaussian Splatting Reconstruction

TL;DR

MuGS is a novel method for view synthesis that combines multi-view stereo and monocular depth cues with Gaussian representations, achieving state-of-the-art results across various scenarios and input configurations.

Contribution

It introduces a generalized Gaussian splatting approach integrating multi-view and monocular features, with a new depth fusion mechanism and reference-view loss for improved reconstruction.

Findings

State-of-the-art performance on DTU, RealEstate10K, LLFF, and Mip-NeRF 360 datasets.

Effective handling of diverse baseline settings, including sparse views and large baselines.

Fast training and inference with 3D Gaussian representations.

Abstract

We present Multi-Baseline Gaussian Splatting (MuGS), a generalized feed-forward approach for novel view synthesis that effectively handles diverse baseline settings, including sparse input views with both small and large baselines. Specifically, we integrate features from Multi-View Stereo (MVS) and Monocular Depth Estimation (MDE) to enhance feature representations for generalizable reconstruction. Next, We propose a projection-and-sampling mechanism for deep depth fusion, which constructs a fine probability volume to guide the regression of the feature map. Furthermore, We introduce a reference-view loss to improve geometry and optimization efficiency. We leverage 3D Gaussian representations to accelerate training and inference time while enhancing rendering quality. MuGS achieves state-of-the-art performance across multiple baseline settings and diverse scenarios ranging from simple objects (DTU) to complex indoor and outdoor scenes (RealEstate10K). We also demonstrate promising zero-shot performance on the LLFF and Mip-NeRF 360 datasets. Code is available at https://github.com/EuclidLou/MuGS.