RobustMedSAM: Degradation-Resilient Medical Image Segmentation via Robust Foundation Model Adaptation

TL;DR

RobustMedSAM enhances medical image segmentation robustness by combining pretrained modules, fine-tuning only the mask decoder across diverse datasets and corruption types, significantly improving performance under degraded conditions.

Contribution

This work introduces a module-wise checkpoint fusion approach, combining pretrained medical and corruption-robust models, with selective fine-tuning for improved resilience in medical image segmentation.

Findings

RobustMedSAM increases Dice score from 0.613 to 0.719 on degraded images.

Structured fusion of pretrained modules effectively improves robustness.

The method performs well across multiple imaging modalities and corruption types.

Abstract

Medical image segmentation models built on Segment Anything Model (SAM) achieve strong performance on clean benchmarks, yet their reliability often degrades under realistic image corruptions such as noise, blur, motion artifacts, and modality-specific distortions. Existing approaches address either medical-domain adaptation or corruption robustness, but not both jointly. In SAM, we find that these capabilities are concentrated in complementary modules: the image encoder preserves medical priors, while the mask decoder governs corruption robustness. Motivated by this observation, we propose RobustMedSAM, which adopts module-wise checkpoint fusion by initializing the image encoder from MedSAM and the mask decoder from RobustSAM under a shared ViT-B architecture. We then fine-tune only the mask decoder on 35 medical datasets from MedSegBench, spanning six imaging modalities and 12 corruption types, while freezing the remaining components to preserve pretrained medical representations. We additionally investigate an SVD-based parameter-efficient variant for limited encoder adaptation. Experiments on both in-distribution and out-of-distribution benchmarks show that RobustMedSAM improves degraded-image Dice from 0.613 to 0.719 (+0.106) over SAM, demonstrating that structured fusion of complementary pretrained models is an effective and practical approach for robust medical image segmentation.