LiDAR Registration with Visual Foundation Models

TL;DR

This paper introduces a novel LiDAR registration method using visual foundation model features from surround-view images, achieving robust 6DoF alignment even with significant domain shifts and without retraining.

Contribution

The authors propose using DINOv2 features from images as point descriptors for LiDAR registration, outperforming complex baselines and not requiring domain-specific retraining.

Findings

Outperforms baseline methods by +24.8 and +17.3 registration recall on NCLT and Oxford datasets.

Handles both sparse LiDAR scans and dense 3D maps effectively.

Does not require retraining or domain-specific adjustments.

Abstract

LiDAR registration is a fundamental task in robotic mapping and localization. A critical component of aligning two point clouds is identifying robust point correspondences using point descriptors. This step becomes particularly challenging in scenarios involving domain shifts, seasonal changes, and variations in point cloud structures. These factors substantially impact both handcrafted and learning-based approaches. In this paper, we address these problems by proposing to use DINOv2 features, obtained from surround-view images, as point descriptors. We demonstrate that coupling these descriptors with traditional registration algorithms, such as RANSAC or ICP, facilitates robust 6DoF alignment of LiDAR scans with 3D maps, even when the map was recorded more than a year before. Although conceptually straightforward, our method substantially outperforms more complex baseline techniques. In contrast to previous learning-based point descriptors, our method does not require domain-specific retraining and is agnostic to the point cloud structure, effectively handling both sparse LiDAR scans and dense 3D maps. We show that leveraging the additional camera data enables our method to outperform the best baseline by +24.8 and +17.3 registration recall on the NCLT and Oxford RobotCar datasets. We publicly release the registration benchmark and the code of our work on https://vfm-registration.cs.uni-freiburg.de.