FermatSyn: SAM2-Enhanced Bidirectional Mamba with Isotropic Spiral Scanning for Multi-Modal Medical Image Synthesis

TL;DR

FermatSyn introduces a novel multi-modal medical image synthesis framework that combines domain-aware anatomical priors, hierarchical feature fusion, and an isotropic spiral scanning strategy to improve global consistency and local detail fidelity.

Contribution

The paper presents FermatSyn, integrating SAM2-based priors, a hierarchical residual downsampling module, and a Fermat spiral scanning approach for enhanced multi-modal medical image synthesis.

Findings

Outperforms state-of-the-art in PSNR, SSIM, FID, and 3D consistency.

Synthesized images enable downstream segmentation comparable to real images.

Reduces directional bias with isotropic receptive fields.

Abstract

Multi-modal medical image synthesis is pivotal for alleviating clinical data scarcity, yet existing methods fail to reconcile global anatomical consistency with high-fidelity local detail. We propose FermatSyn, which addresses three persistent limitations: (1)~a SAM2-based Prior Encoder that injects domain-aware anatomical knowledge via Lo-RA$^{+}$ efficient fine-tuning of a frozen SAM2 Vision Transformer; (2)~a Hierarchical Residual Downsampling Module (HRDM) coupled with a Cross-scale Integration Network (CIN) that preserves high-frequency lesion details and adaptively fuses global--local representations; and (3)~a continuity constrained Fermat Spiral Scanning strategy within a Bidirectional Fermat Scan Mamba (BFS-Mamba), constructing an approximately isotropic receptive field that substantially reduces the directional bias of raster or spiral serialization. Experiments on SynthRAD2023, BraTS2019, BraTS-MEN, and BraTS-MET show FermatSyn surpasses state-of-the-art methods in PSNR, SSIM, FID, and 3D structural consistency. Downstream segmentation on synthesized images yields no significant difference from real-image training ($p{>}0.05$), confirming clinical utility. Code will be released upon publication. \keywords{Medical image synthesis \and SAM2 \and Mamba \and Fermat spiral scanning \and Anatomical prior \and Cross-modal}