Visual Localization in 3D Maps: Comparing Point Cloud, Mesh, and NeRF Representations

TL;DR

This paper introduces a unified visual localization system that effectively localizes camera images across point cloud, mesh, and NeRF 3D map representations, demonstrating high success rates and real-time performance in diverse environments.

Contribution

The paper presents a novel, unified localization approach capable of operating across different 3D map types using synthetic views and learning-based descriptors, outperforming traditional SfM methods.

Findings

All map types achieve over 55% success rate in localization.

NeRF-based images yield an average success rate of 72%.

The system operates at 1Hz in real-time on a GPU-equipped laptop.

Abstract

Recent advances in mapping techniques have enabled the creation of highly accurate dense 3D maps during robotic missions, such as point clouds, meshes, or NeRF-based representations. These developments present new opportunities for reusing these maps for localization. However, there remains a lack of a unified approach that can operate seamlessly across different map representations. This paper presents and evaluates a global visual localization system capable of localizing a single camera image across various 3D map representations built using both visual and lidar sensing. Our system generates a database by synthesizing novel views of the scene, creating RGB and depth image pairs. Leveraging the precise 3D geometric map, our method automatically defines rendering poses, reducing the number of database images while preserving retrieval performance. To bridge the domain gap between real query camera images and synthetic database images, our approach utilizes learning-based descriptors and feature detectors. We evaluate the system's performance through extensive real-world experiments conducted in both indoor and outdoor settings, assessing the effectiveness of each map representation and demonstrating its advantages over traditional structure-from-motion (SfM) localization approaches. The results show that all three map representations can achieve consistent localization success rates of 55% and higher across various environments. NeRF synthesized images show superior performance, localizing query images at an average success rate of 72%. Furthermore, we demonstrate an advantage over SfM-based approaches that our synthesized database enables localization in the reverse travel direction which is unseen during the mapping process. Our system, operating in real-time on a mobile laptop equipped with a GPU, achieves a processing rate of 1Hz.