Hg-I2P: Bridging Modalities for Generalizable Image-to-Point-Cloud Registration via Heterogeneous Graphs

TL;DR

Hg-I2P introduces a heterogeneous graph approach to improve image-to-point-cloud registration by enhancing feature interaction and correspondence pruning, leading to better generalization and accuracy across diverse scenarios.

Contribution

The paper proposes a novel heterogeneous graph framework for I2P registration that refines features and correspondences within a unified architecture, improving robustness and performance.

Findings

Outperforms existing methods on six benchmarks in accuracy.

Enhances feature discriminability through cross-modal interaction.

Effectively prunes unreliable correspondences using graph-based consistency.

Abstract

Image-to-point-cloud (I2P) registration aims to align 2D images with 3D point clouds by establishing reliable 2D-3D correspondences. The drastic modality gap between images and point clouds makes it challenging to learn features that are both discriminative and generalizable, leading to severe performance drops in unseen scenarios. We address this challenge by introducing a heterogeneous graph that enables refining both cross-modal features and correspondences within a unified architecture. The proposed graph represents a mapping between segmented 2D and 3D regions, which enhances cross-modal feature interaction and thus improves feature discriminability. In addition, modeling the consistency among vertices and edges within the graph enables pruning of unreliable correspondences. Building on these insights, we propose a heterogeneous graph embedded I2P registration method, termed Hg-I2P. It learns a heterogeneous graph by mining multi-path feature relationships, adapts features under the guidance of heterogeneous edges, and prunes correspondences using graph-based projection consistency. Experiments on six indoor and outdoor benchmarks under cross-domain setups demonstrate that Hg-I2P significantly outperforms existing methods in both generalization and accuracy. Code is released on https://github.com/anpei96/hg-i2p-demo.