NeVStereo: A NeRF-Driven NVS-Stereo Architecture for High-Fidelity 3D Tasks

TL;DR

NeVStereo is a novel neural rendering framework that jointly estimates camera poses, depth, and high-fidelity view synthesis from casual multi-view RGB images, improving 3D reconstruction quality.

Contribution

It introduces a NeRF-driven stereo-aware architecture with confidence-guided depth, pose refinement, and iterative updates, addressing common NeRF issues and enabling accurate 3D tasks.

Findings

Achieves up to 36% lower depth error

Improves pose accuracy by 10.4%

Enhances NVS fidelity and mesh quality

Abstract

In modern dense 3D reconstruction, feed-forward systems (e.g., VGGT, pi3) focus on end-to-end matching and geometry prediction but do not explicitly output the novel view synthesis (NVS). Neural rendering-based approaches offer high-fidelity NVS and detailed geometry from posed images, yet they typically assume fixed camera poses and can be sensitive to pose errors. As a result, it remains non-trivial to obtain a single framework that can offer accurate poses, reliable depth, high-quality rendering, and accurate 3D surfaces from casually captured views. We present NeVStereo, a NeRF-driven NVS-stereo architecture that aims to jointly deliver camera poses, multi-view depth, novel view synthesis, and surface reconstruction from multi-view RGB-only inputs. NeVStereo combines NeRF-based NVS for stereo-friendly renderings, confidence-guided multi-view depth estimation, NeRF-coupled bundle adjustment for pose refinement, and an iterative refinement stage that updates both depth and the radiance field to improve geometric consistency. This design mitigated the common NeRF-based issues such as surface stacking, artifacts, and pose-depth coupling. Across indoor, outdoor, tabletop, and aerial benchmarks, our experiments indicate that NeVStereo achieves consistently strong zero-shot performance, with up to 36% lower depth error, 10.4% improved pose accuracy, 4.5% higher NVS fidelity, and state-of-the-art mesh quality (F1 91.93%, Chamfer 4.35 mm) compared to existing prestigious methods.