LLaMA-Reg: Using LLaMA 2 for Unsupervised Medical Image Registration

TL;DR

This paper introduces a novel medical image registration method leveraging pretrained large language models to enhance registration accuracy, demonstrating state-of-the-art results on MRI datasets.

Contribution

It proposes a new approach that uses a frozen pretrained language model with an adapter for deep feature encoding in medical image registration.

Findings

Achieved state-of-the-art registration accuracy on knee and brain MRI datasets.

Effectively improved registration performance by integrating deep features from a large language model.

Demonstrated the potential of language models in medical image analysis tasks.

Abstract

Medical image registration is an essential topic in medical image analysis. In this paper, we propose a method for medical image registration using a pretrained large language model. We find that using the pretrained large language model to encode deep features of the medical images in the registration model can effectively improve image registration accuracy, indicating the great potential of the large language model in medical image registration tasks. We use dual encoders to perform deep feature extraction on image pairs and then input the features into the pretrained large language model. To adapt the large language model to our registration task, the weights of the large language model are frozen in the registration model, and an adapter is utilized to fine-tune the large language model, which aims at (a) mapping the visual tokens to the language space before the large language model computing, (b) project the modeled language tokens output from the large language model to the visual space. Our method combines output features from the fine-tuned large language model with the features output from each encoder layer to gradually generate the deformation fields required for registration in the decoder. To demonstrate the effectiveness of the large prediction model in registration tasks, we conducted experiments on knee and brain MRI and achieved state-of-the-art results.