TranSplat: Generalizable 3D Gaussian Splatting from Sparse Multi-View Images with Transformers

TL;DR

TranSplat is a novel 3D Gaussian Splatting method that improves multi-view feature matching accuracy using depth confidence and monocular depth priors, achieving state-of-the-art results in sparse-view 3D reconstruction.

Contribution

It introduces a new strategy combining depth confidence maps and monocular depth priors to enhance 3D Gaussian Splatting from sparse multi-view images.

Findings

Achieves top performance on RealEstate10K and ACID benchmarks.

Maintains competitive speed and strong cross-dataset generalization.

Outperforms existing G-3DGS methods in sparse-view scenarios.

Abstract

Compared with previous 3D reconstruction methods like Nerf, recent Generalizable 3D Gaussian Splatting (G-3DGS) methods demonstrate impressive efficiency even in the sparse-view setting. However, the promising reconstruction performance of existing G-3DGS methods relies heavily on accurate multi-view feature matching, which is quite challenging. Especially for the scenes that have many non-overlapping areas between various views and contain numerous similar regions, the matching performance of existing methods is poor and the reconstruction precision is limited. To address this problem, we develop a strategy that utilizes a predicted depth confidence map to guide accurate local feature matching. In addition, we propose to utilize the knowledge of existing monocular depth estimation models as prior to boost the depth estimation precision in non-overlapping areas between views. Combining the proposed strategies, we present a novel G-3DGS method named TranSplat, which obtains the best performance on both the RealEstate10K and ACID benchmarks while maintaining competitive speed and presenting strong cross-dataset generalization ability. Our code, and demos will be available at: https://xingyoujun.github.io/transplat.