NAP3D: NeRF Assisted 3D-3D Pose Alignment for Autonomous Vehicles

TL;DR

NAP3D introduces a novel 3D-3D pose alignment method using NeRFs to improve autonomous vehicle localization, especially when revisiting locations is not feasible, achieving high accuracy and robustness.

Contribution

This work presents NAP3D, a new NeRF-assisted 3D-3D pose alignment technique that refines pose estimates without requiring revisited locations, enhancing localization robustness.

Findings

Achieves camera pose correction within 5 cm on a custom dataset.

Outperforms 2D-3D PnP baseline in 3D alignment RMSE by ~6 cm.

Maintains robustness across varying noise levels.

Abstract

Accurate localization is essential for autonomous vehicles, yet sensor noise and drift over time can lead to significant pose estimation errors, particularly in long-horizon environments. A common strategy for correcting accumulated error is visual loop closure in SLAM, which adjusts the pose graph when the agent revisits previously mapped locations. These techniques typically rely on identifying visual mappings between the current view and previously observed scenes and often require fusing data from multiple sensors. In contrast, this work introduces NeRF-Assisted 3D-3D Pose Alignment (NAP3D), a complementary approach that leverages 3D-3D correspondences between the agent's current depth image and a pre-trained Neural Radiance Field (NeRF). By directly aligning 3D points from the observed scene with synthesized points from the NeRF, NAP3D refines the estimated pose even from novel viewpoints, without relying on revisiting previously observed locations. This robust 3D-3D formulation provides advantages over conventional 2D-3D localization methods while remaining comparable in accuracy and applicability. Experiments demonstrate that NAP3D achieves camera pose correction within 5 cm on a custom dataset, robustly outperforming a 2D-3D Perspective-N-Point baseline. On TUM RGB-D, NAP3D consistently improves 3D alignment RMSE by approximately 6 cm compared to this baseline given varying noise, despite PnP achieving lower raw rotation and translation parameter error in some regimes, highlighting NAP3D's improved geometric consistency in 3D space. By providing a lightweight, dataset-agnostic tool, NAP3D complements existing SLAM and localization pipelines when traditional loop closure is unavailable.