CMR-Agent: Learning a Cross-Modal Agent for Iterative Image-to-Point Cloud Registration

TL;DR

This paper introduces CMR-Agent, a reinforcement learning-based approach for iterative image-to-point cloud registration that improves accuracy and efficiency by reformulating the task as a Markov decision process and utilizing a hybrid state representation.

Contribution

It proposes a novel cross-modal registration agent using reinforcement learning and a hybrid state representation to enhance iterative registration accuracy and efficiency.

Findings

Achieves competitive accuracy on KITTI-Odometry and NuScenes datasets.

Each registration iteration takes only a few milliseconds after initial embedding.

Outperforms existing methods in accuracy and computational efficiency.

Abstract

Image-to-point cloud registration aims to determine the relative camera pose of an RGB image with respect to a point cloud. It plays an important role in camera localization within pre-built LiDAR maps. Despite the modality gaps, most learning-based methods establish 2D-3D point correspondences in feature space without any feedback mechanism for iterative optimization, resulting in poor accuracy and interpretability. In this paper, we propose to reformulate the registration procedure as an iterative Markov decision process, allowing for incremental adjustments to the camera pose based on each intermediate state. To achieve this, we employ reinforcement learning to develop a cross-modal registration agent (CMR-Agent), and use imitation learning to initialize its registration policy for stability and quick-start of the training. According to the cross-modal observations, we propose a 2D-3D hybrid state representation that fully exploits the fine-grained features of RGB images while reducing the useless neutral states caused by the spatial truncation of camera frustum. Additionally, the overall framework is well-designed to efficiently reuse one-shot cross-modal embeddings, avoiding repetitive and time-consuming feature extraction. Extensive experiments on the KITTI-Odometry and NuScenes datasets demonstrate that CMR-Agent achieves competitive accuracy and efficiency in registration. Once the one-shot embeddings are completed, each iteration only takes a few milliseconds.