Cross-modal Attention for MRI and Ultrasound Volume Registration

TL;DR

This paper introduces a novel cross-modal attention mechanism for MRI and ultrasound volume registration, significantly enhancing registration accuracy and interpretability over traditional CNN approaches.

Contribution

It develops a specialized self-attention block for cross-modal image registration, outperforming larger CNN models and improving interpretability.

Findings

Cross-modal attention block improves registration accuracy

Model outperforms larger CNN networks by 10 times in size

Visualization techniques enhance interpretability

Abstract

Prostate cancer biopsy benefits from accurate fusion of transrectal ultrasound (TRUS) and magnetic resonance (MR) images. In the past few years, convolutional neural networks (CNNs) have been proved powerful in extracting image features crucial for image registration. However, challenging applications and recent advances in computer vision suggest that CNNs are quite limited in its ability to understand spatial correspondence between features, a task in which the self-attention mechanism excels. This paper aims to develop a self-attention mechanism specifically for cross-modal image registration. Our proposed cross-modal attention block effectively maps each of the features in one volume to all features in the corresponding volume. Our experimental results demonstrate that a CNN network designed with the cross-modal attention block embedded outperforms an advanced CNN network 10 times of its size. We also incorporated visualization techniques to improve the interpretability of our network. The source code of our work is available at https://github.com/DIAL-RPI/Attention-Reg .