SparSplat: Fast Multi-View Reconstruction with Generalizable 2D Gaussian Splatting

TL;DR

This paper introduces SparSplat, a fast and generalizable multi-view reconstruction method that jointly performs sparse 3D shape reconstruction and novel view synthesis using 2D Gaussian splatting, achieving state-of-the-art results and high speed.

Contribution

It proposes a novel MVS-based pipeline that regresses 2D Gaussian surface parameters for joint 3D reconstruction and NVS, outperforming prior methods in accuracy and speed.

Findings

Achieves state-of-the-art Chamfer distance on DTU benchmark.

Demonstrates strong generalization on multiple datasets.

Offers nearly 100x faster inference than previous methods.

Abstract

Recovering 3D information from scenes via multi-view stereo reconstruction (MVS) and novel view synthesis (NVS) is inherently challenging, particularly in scenarios involving sparse-view setups. The advent of 3D Gaussian Splatting (3DGS) enabled real-time, photorealistic NVS. Following this, 2D Gaussian Splatting (2DGS) leveraged perspective accurate 2D Gaussian primitive rasterization to achieve accurate geometry representation during rendering, improving 3D scene reconstruction while maintaining real-time performance. Recent approaches have tackled the problem of sparse real-time NVS using 3DGS within a generalizable, MVS-based learning framework to regress 3D Gaussian parameters. Our work extends this line of research by addressing the challenge of generalizable sparse 3D reconstruction and NVS jointly, and manages to perform successfully at both tasks. We propose an MVS-based learning pipeline that regresses 2DGS surface element parameters in a feed-forward fashion to perform 3D shape reconstruction and NVS from sparse-view images. We further show that our generalizable pipeline can benefit from preexisting foundational multi-view deep visual features. The resulting model attains the state-of-the-art results on the DTU sparse 3D reconstruction benchmark in terms of Chamfer distance to ground-truth, as-well as state-of-the-art NVS. It also demonstrates strong generalization on the BlendedMVS and Tanks and Temples datasets. We note that our model outperforms the prior state-of-the-art in feed-forward sparse view reconstruction based on volume rendering of implicit representations, while offering an almost 2 orders of magnitude higher inference speed.