Continual Alignment for SAM: Rethinking Foundation Models for Medical Image Segmentation in Continual Learning

TL;DR

This paper introduces CA-SAM, a continual learning method that adapts the Segment Anything Model for medical image segmentation, balancing performance and efficiency while mitigating catastrophic forgetting across multiple datasets.

Contribution

It proposes the Alignment Layer and CA-SAM strategy, enabling SAM to be effectively used in continual learning for medical images with improved accuracy and reduced computation.

Findings

Achieves state-of-the-art performance on nine datasets

Effectively mitigates catastrophic forgetting in continual learning

Balances accuracy and computational efficiency

Abstract

In medical image segmentation, heterogeneous privacy policies across institutions often make joint training on pooled datasets infeasible, motivating continual image segmentation-learning from data streams without catastrophic forgetting. While the Segment Anything Model (SAM) offers strong zero-shot priors and has been widely fine-tuned across downstream tasks, its large parameter count and computational overhead challenge practical deployment. This paper demonstrates that the SAM paradigm is highly promising once its computational efficiency and performance can be balanced. To this end, we introduce the Alignment Layer, a lightweight, plug-and-play module which aligns encoder-decoder feature distributions to efficiently adapt SAM to specific medical images, improving accuracy while reducing computation. Building on SAM and the Alignment Layer, we then propose Continual Alignment for SAM (CA-SAM), a continual learning strategy that automatically adapts the appropriate Alignment Layer to mitigate catastrophic forgetting, while leveraging SAM's zero-shot priors to preserve strong performance on unseen medical datasets. Experimented across nine medical segmentation datasets under continual-learning scenario, CA-SAM achieves state-of-the-art performance. Our code, models and datasets will be released on \mbox{https://github.com/azzzzyo/Continual-Alignment-for-SAM.}