Synthetic Volumetric Data Generation Enables Zero-Shot Generalization of Foundation Models in 3D Medical Image Segmentation

TL;DR

This paper introduces SynthFM-3D, a synthetic data generation framework that enhances zero-shot generalization of foundation models like SAM 2 in 3D medical image segmentation across various modalities and structures.

Contribution

SynthFM-3D is the first analytical framework to model 3D variability for synthetic data generation, improving foundation model performance without real annotations.

Findings

SynthFM-3D training improves Dice scores significantly over baseline.

SynthFM-3D outperforms supervised models on unseen cardiac ultrasound data.

Zero-shot segmentation performance is substantially enhanced across multiple modalities.

Abstract

Foundation models such as Segment Anything Model 2 (SAM 2) exhibit strong generalization on natural images and videos but perform poorly on medical data due to differences in appearance statistics, imaging physics, and three-dimensional structure. To address this gap, we introduce SynthFM-3D, an analytical framework that mathematically models 3D variability in anatomy, contrast, boundary definition, and noise to generate synthetic data for training promptable segmentation models without real annotations. We fine-tuned SAM 2 on 10,000 SynthFM-3D volumes and evaluated it on eleven anatomical structures across three medical imaging modalities (CT, MR, ultrasound) from five public datasets. SynthFM-3D training led to consistent and statistically significant Dice score improvements over the pretrained SAM 2 baseline, demonstrating stronger zero-shot generalization across modalities. When compared with the supervised SAM-Med3D model on unseen cardiac ultrasound data, SynthFM-3D achieved 2-3x higher Dice scores, establishing analytical 3D data modeling as an effective pathway to modality-agnostic medical segmentation.