Structural Attention: Rethinking Transformer for Unpaired Medical Image Synthesis

TL;DR

This paper introduces UNest, a structural attention-based transformer architecture that leverages anatomical priors to improve unpaired medical image synthesis across multiple modalities, outperforming existing methods.

Contribution

We propose UNest, a novel transformer architecture with structural inductive biases and anatomical guidance for unpaired medical image synthesis.

Findings

UNest improves synthesis quality by up to 19.30% over recent methods.

Incorporating structural biases enhances anatomical detail preservation.

The method is effective across MR, CT, and PET datasets.

Abstract

Unpaired medical image synthesis aims to provide complementary information for an accurate clinical diagnostics, and address challenges in obtaining aligned multi-modal medical scans. Transformer-based models excel in imaging translation tasks thanks to their ability to capture long-range dependencies. Although effective in supervised training settings, their performance falters in unpaired image synthesis, particularly in synthesizing structural details. This paper empirically demonstrates that, lacking strong inductive biases, Transformer can converge to non-optimal solutions in the absence of paired data. To address this, we introduce UNet Structured Transformer (UNest), a novel architecture incorporating structural inductive biases for unpaired medical image synthesis. We leverage the foundational Segment-Anything Model to precisely extract the foreground structure and perform structural attention within the main anatomy. This guides the model to learn key anatomical regions, thus improving structural synthesis under the lack of supervision in unpaired training. Evaluated on two public datasets, spanning three modalities, i.e., MR, CT, and PET, UNest improves recent methods by up to 19.30% across six medical image synthesis tasks. Our code is released at https://github.com/HieuPhan33/MICCAI2024-UNest.