Improved Baselines with Synchronized Encoding for Universal Medical Image Segmentation

TL;DR

SyncSAM introduces a synchronized dual-branch encoder and multi-scale decoder to improve medical image segmentation, achieving state-of-the-art results and strong zero-shot performance across diverse datasets.

Contribution

The paper proposes SyncSAM, a novel architecture combining convolution and Transformer features with synchronized encoding for enhanced medical image segmentation.

Findings

Achieves state-of-the-art segmentation performance on test datasets.

Exhibits strong zero-shot generalization to unseen datasets.

Provides a new strong baseline for future medical segmentation research.

Abstract

Large foundation models, known for their strong zero-shot generalization capabilities, can be applied to a wide range of downstream tasks. However, developing foundation models for medical image segmentation poses a significant challenge due to the domain gap between natural and medical images. While fine-tuning techniques based on the Segment Anything Model (SAM) have been explored, they primarily focus on scaling up data or refining inference strategies without incorporating domain-specific architectural designs, limiting their zero-shot performance. To optimize segmentation performance under standard inference settings and provide a strong baseline for future research, we introduce SyncSAM, which employs a synchronized dual-branch encoder that integrates convolution and Transformer features in a synchronized manner to enhance medical image encoding, and a multi-scale dual-branch decoder to preserve image details. SyncSAM is trained on two of the largest medical image segmentation datasets, SA-Med2D-20M and IMed-361M, resulting in a series of pre-trained models for universal medical image segmentation. Experimental results demonstrate that SyncSAM not only achieves state-of-the-art performance on test sets but also exhibits strong zero-shot capabilities on unseen datasets. Code and checkpoints are available at https://github.com/Hhankyangg/SyncSAM.