Joint Learning of Depth, Pose, and Local Radiance Field for Large Scale Monocular 3D Reconstruction

TL;DR

This paper presents a joint learning framework that integrates depth, pose, and radiance field estimation to enable large-scale, photorealistic 3D reconstruction from monocular videos, overcoming scale ambiguity and drift issues.

Contribution

The authors introduce a novel system combining a ViT-based depth network, feature-based bundle adjustment, and incremental local radiance fields for scalable, drift-free 3D reconstruction.

Findings

Achieves up to 18x lower trajectory error than previous methods.

Enables city-scale scene coverage on a single GPU.

Maintains high-fidelity view synthesis from monocular videos.

Abstract

Photorealistic 3-D reconstruction from monocular video collapses in large-scale scenes when depth, pose, and radiance are solved in isolation: scale-ambiguous depth yields ghost geometry, long-horizon pose drift corrupts alignment, and a single global NeRF cannot model hundreds of metres of content. We introduce a joint learning framework that couples all three factors and demonstrably overcomes each failure case. Our system begins with a Vision-Transformer (ViT) depth network trained with metric-scale supervision, giving globally consistent depths despite wide field-of-view variations. A multi-scale feature bundle-adjustment (BA) layer refines camera poses directly in feature space--leveraging learned pyramidal descriptors instead of brittle keypoints--to suppress drift on unconstrained trajectories. For scene representation, we deploy an incremental local-radiance-field hierarchy: new hash-grid NeRFs are allocated and frozen on-the-fly when view overlap falls below a threshold, enabling city-block-scale coverage on a single GPU. Evaluated on the Tanks and Temples benchmark, our method reduces Absolute Trajectory Error to 0.001-0.021 m across eight indoor-outdoor sequences--up to 18x lower than BARF and 2x lower than NoPe-NeRF--while maintaining sub-pixel Relative Pose Error. These results demonstrate that metric-scale, drift-free 3-D reconstruction and high-fidelity novel-view synthesis are achievable from a single uncalibrated RGB camera.