TFCT-I2P: Three stream fusion network with color aware transformer for image-to-point cloud registration

TL;DR

This paper introduces TFCT-I2P, a novel three-stream fusion network with a color-aware transformer that effectively aligns features from images and point clouds, improving registration accuracy across multiple datasets.

Contribution

The paper proposes a new multi-modal registration method combining a three-stream fusion network and a color-aware transformer to enhance image-to-point-cloud alignment.

Findings

Outperforms state-of-the-art in Inlier Ratio by 1.5%

Achieves 0.4% higher Feature Matching Recall

Improves Registration Recall by 5.4%

Abstract

Along with the advancements in artificial intelligence technologies, image-to-point-cloud registration (I2P) techniques have made significant strides. Nevertheless, the dimensional differences in the features of points cloud (three-dimension) and image (two-dimension) continue to pose considerable challenges to their development. The primary challenge resides in the inability to leverage the features of one modality to augment those of another, thereby complicating the alignment of features within the latent space. To address this challenge, we propose an image-to-point-cloud method named as TFCT-I2P. Initially, we introduce a Three-Stream Fusion Network (TFN), which integrates color information from images with structural information from point clouds, facilitating the alignment of features from both modalities. Subsequently, to effectively mitigate patch-level misalignments introduced by the inclusion of color information, we design a Color-Aware Transformer (CAT). Finally, we conduct extensive experiments on 7Scenes, RGB-D Scenes V2, ScanNet V2, and a self-collected dataset. The results demonstrate that TFCT-I2P surpasses state-of-the-art methods by 1.5% in Inlier Ratio, 0.4% in Feature Matching Recall, and 5.4% in Registration Recall. Therefore, we believe that the proposed TFCT-I2P contributes to the advancement of I2P registration.