Uncertainty-guided Optimal Transport in Depth Supervised Sparse-View 3D Gaussian

TL;DR

This paper introduces UGOT, a method that improves sparse-view 3D reconstruction and view synthesis by incorporating depth uncertainty and optimal transport strategies, outperforming existing approaches.

Contribution

The paper proposes a novel depth supervision method using uncertainty-aware priors and patch-wise optimal transport to enhance 3D Gaussian-based view synthesis from sparse inputs.

Findings

Outperforms state-of-the-art methods on LLFF, DTU, and Blender datasets.

Achieves superior novel view synthesis quality.

Effectively handles depth prediction uncertainty in 3D reconstruction.

Abstract

3D Gaussian splatting has demonstrated impressive performance in real-time novel view synthesis. However, achieving successful reconstruction from RGB images generally requires multiple input views captured under static conditions. To address the challenge of sparse input views, previous approaches have incorporated depth supervision into the training of 3D Gaussians to mitigate overfitting, using dense predictions from pretrained depth networks as pseudo-ground truth. Nevertheless, depth predictions from monocular depth estimation models inherently exhibit significant uncertainty in specific areas. Relying solely on pixel-wise L2 loss may inadvertently incorporate detrimental noise from these uncertain areas. In this work, we introduce a novel method to supervise the depth distribution of 3D Gaussians, utilizing depth priors with integrated uncertainty estimates. To address these localized errors in depth predictions, we integrate a patch-wise optimal transport strategy to complement traditional L2 loss in depth supervision. Extensive experiments conducted on the LLFF, DTU, and Blender datasets demonstrate that our approach, UGOT, achieves superior novel view synthesis and consistently outperforms state-of-the-art methods.