SPARS3R: Semantic Prior Alignment and Regularization for Sparse 3D Reconstruction

TL;DR

SPARS3R introduces a novel method combining pose estimation and dense point cloud alignment to enable photorealistic 3D reconstruction from sparse views, outperforming existing approaches.

Contribution

It proposes a two-step alignment process integrating Structure-from-Motion and semantic outlier correction for improved sparse-view 3D reconstruction.

Findings

Achieves photorealistic rendering with sparse images.

Significantly outperforms existing methods in accuracy.

Effective in handling outliers through semantic alignment.

Abstract

Recent efforts in Gaussian-Splat-based Novel View Synthesis can achieve photorealistic rendering; however, such capability is limited in sparse-view scenarios due to sparse initialization and over-fitting floaters. Recent progress in depth estimation and alignment can provide dense point cloud with few views; however, the resulting pose accuracy is suboptimal. In this work, we present SPARS3R, which combines the advantages of accurate pose estimation from Structure-from-Motion and dense point cloud from depth estimation. To this end, SPARS3R first performs a Global Fusion Alignment process that maps a prior dense point cloud to a sparse point cloud from Structure-from-Motion based on triangulated correspondences. RANSAC is applied during this process to distinguish inliers and outliers. SPARS3R then performs a second, Semantic Outlier Alignment step, which extracts semantically coherent regions around the outliers and performs local alignment in these regions. Along with several improvements in the evaluation process, we demonstrate that SPARS3R can achieve photorealistic rendering with sparse images and significantly outperforms existing approaches.