JointSplat: Probabilistic Joint Flow-Depth Optimization for Sparse-View Gaussian Splatting

TL;DR

JointSplat introduces a probabilistic framework that optimally combines optical flow and depth information for improved sparse-view 3D scene reconstruction, addressing limitations of existing methods in low-texture and uncertain regions.

Contribution

It proposes a novel probabilistic optimization mechanism and a multi-view depth-consistency loss to enhance flow-depth joint estimation in sparse-view 3D reconstruction.

Findings

Outperforms state-of-the-art methods on RealEstate10K and ACID datasets.

Improves robustness and accuracy in low-texture and ambiguous regions.

Effectively leverages flow-depth complementarity through probabilistic scaling.

Abstract

Reconstructing 3D scenes from sparse viewpoints is a long-standing challenge with wide applications. Recent advances in feed-forward 3D Gaussian sparse-view reconstruction methods provide an efficient solution for real-time novel view synthesis by leveraging geometric priors learned from large-scale multi-view datasets and computing 3D Gaussian centers via back-projection. Despite offering strong geometric cues, both feed-forward multi-view depth estimation and flow-depth joint estimation face key limitations: the former suffers from mislocation and artifact issues in low-texture or repetitive regions, while the latter is prone to local noise and global inconsistency due to unreliable matches when ground-truth flow supervision is unavailable. To overcome this, we propose JointSplat, a unified framework that leverages the complementarity between optical flow and depth via a novel probabilistic optimization mechanism. Specifically, this pixel-level mechanism scales the information fusion between depth and flow based on the matching probability of optical flow during training. Building upon the above mechanism, we further propose a novel multi-view depth-consistency loss to leverage the reliability of supervision while suppressing misleading gradients in uncertain areas. Evaluated on RealEstate10K and ACID, JointSplat consistently outperforms state-of-the-art (SOTA) methods, demonstrating the effectiveness and robustness of our proposed probabilistic joint flow-depth optimization approach for high-fidelity sparse-view 3D reconstruction.