RegCL: Continual Adaptation of Segment Anything Model via Model Merging

TL;DR

RegCL introduces a model merging framework for continual learning that consolidates multi-domain knowledge in the Segment Anything Model without increasing model size or requiring historical data, addressing catastrophic forgetting.

Contribution

It proposes a novel non-replay continual learning method using model merging to efficiently adapt SAM across multiple domains.

Findings

Achieves effective multi-domain knowledge integration.

Maintains constant model size regardless of tasks.

Outperforms existing methods in continual learning scenarios.

Abstract

To address the performance limitations of the Segment Anything Model (SAM) in specific domains, existing works primarily adopt adapter-based one-step adaptation paradigms. However, some of these methods are specific developed for specific domains. If used on other domains may lead to performance degradation. This issue of catastrophic forgetting severely limits the model's scalability. To address this issue, this paper proposes RegCL, a novel non-replay continual learning (CL) framework designed for efficient multi-domain knowledge integration through model merging. Specifically, RegCL incorporates the model merging algorithm into the continual learning paradigm by merging the parameters of SAM's adaptation modules (e.g., LoRA modules) trained on different domains. The merging process is guided by weight optimization, which minimizes prediction discrepancies between the merged model and each of the domain-specific models. RegCL effectively consolidates multi-domain knowledge while maintaining parameter efficiency, i.e., the model size remains constant regardless of the number of tasks, and no historical data storage is required. Experimental results demonstrate that RegCL achieves favorable continual learning performance across multiple downstream datasets, validating its effectiveness in dynamic scenarios.