VDNeRF: Vision-only Dynamic Neural Radiance Field for Urban Scenes

TL;DR

VDNeRF is a novel vision-only approach that jointly reconstructs dynamic urban scenes and estimates camera trajectories without extra pose data, outperforming existing methods in pose accuracy and view synthesis.

Contribution

It introduces a dual-NeRF framework that self-supervisedly separates static and dynamic scene components and accurately recovers camera poses without additional sensors.

Findings

Outperforms state-of-the-art pose-free NeRF methods in urban scenes

Accurately estimates camera trajectories without external pose data

Effectively reconstructs dynamic objects and backgrounds in large-scale scenes

Abstract

Neural Radiance Fields (NeRFs) implicitly model continuous three-dimensional scenes using a set of images with known camera poses, enabling the rendering of photorealistic novel views. However, existing NeRF-based methods encounter challenges in applications such as autonomous driving and robotic perception, primarily due to the difficulty of capturing accurate camera poses and limitations in handling large-scale dynamic environments. To address these issues, we propose Vision-only Dynamic NeRF (VDNeRF), a method that accurately recovers camera trajectories and learns spatiotemporal representations for dynamic urban scenes without requiring additional camera pose information or expensive sensor data. VDNeRF employs two separate NeRF models to jointly reconstruct the scene. The static NeRF model optimizes camera poses and static background, while the dynamic NeRF model incorporates the 3D scene flow to ensure accurate and consistent reconstruction of dynamic objects. To address the ambiguity between camera motion and independent object motion, we design an effective and powerful training framework to achieve robust camera pose estimation and self-supervised decomposition of static and dynamic elements in a scene. Extensive evaluations on mainstream urban driving datasets demonstrate that VDNeRF surpasses state-of-the-art NeRF-based pose-free methods in both camera pose estimation and dynamic novel view synthesis.