Guiding Registration with Emergent Similarity from Pre-Trained Diffusion Models

TL;DR

This paper introduces a novel image registration method that uses features from pre-trained diffusion models to identify semantic correspondences, improving alignment accuracy in challenging medical imaging scenarios.

Contribution

It leverages diffusion model features as a semantic similarity measure to guide deformable registration, addressing limitations of intensity-based methods.

Findings

Outperforms traditional intensity-based registration in challenging cases

Successfully applies to multimodal 2D and monomodal 3D medical images

Demonstrates improved anatomical alignment accuracy

Abstract

Diffusion models, while trained for image generation, have emerged as powerful foundational feature extractors for downstream tasks. We find that off-the-shelf diffusion models, trained exclusively to generate natural RGB images, can identify semantically meaningful correspondences in medical images. Building on this observation, we propose to leverage diffusion model features as a similarity measure to guide deformable image registration networks. We show that common intensity-based similarity losses often fail in challenging scenarios, such as when certain anatomies are visible in one image but absent in another, leading to anatomically inaccurate alignments. In contrast, our method identifies true semantic correspondences, aligning meaningful structures while disregarding those not present across images. We demonstrate superior performance of our approach on two tasks: multimodal 2D registration (DXA to X-Ray) and monomodal 3D registration (brain-extracted to non-brain-extracted MRI). Code: https://github.com/uncbiag/dgir