Tell2Reg: Establishing spatial correspondence between images by the same language prompts

TL;DR

Tell2Reg introduces a fully automated, training-free image registration method that leverages large multimodal models and language prompts to establish spatial correspondence, demonstrated on challenging prostate MRI registration tasks.

Contribution

The paper presents a novel registration approach using language prompts and pre-trained models, eliminating the need for training data and outperforming unsupervised methods.

Findings

Outperforms unsupervised registration methods

Comparable to weakly-supervised methods

Reveals potential link between language semantics and spatial correspondence

Abstract

Spatial correspondence can be represented by pairs of segmented regions, such that the image registration networks aim to segment corresponding regions rather than predicting displacement fields or transformation parameters. In this work, we show that such a corresponding region pair can be predicted by the same language prompt on two different images using the pre-trained large multimodal models based on GroundingDINO and SAM. This enables a fully automated and training-free registration algorithm, potentially generalisable to a wide range of image registration tasks. In this paper, we present experimental results using one of the challenging tasks, registering inter-subject prostate MR images, which involves both highly variable intensity and morphology between patients. Tell2Reg is training-free, eliminating the need for costly and time-consuming data curation and labelling that was previously required for this registration task. This approach outperforms unsupervised learning-based registration methods tested, and has a performance comparable to weakly-supervised methods. Additional qualitative results are also presented to suggest that, for the first time, there is a potential correlation between language semantics and spatial correspondence, including the spatial invariance in language-prompted regions and the difference in language prompts between the obtained local and global correspondences. Code is available at https://github.com/yanwenCi/Tell2Reg.git.