Pruned Adaptation Modules: A Simple yet Strong Baseline for Continual Foundation Models

TL;DR

This paper introduces Pruned Adaptation Modules (PAM), a lightweight, scalable method for continual learning with foundation models that reduces parameters and mitigates forgetting, serving as a strong baseline for future research.

Contribution

PAM is a simple, effective baseline that significantly reduces trainable parameters and outperforms existing FM-based continual learning methods, bridging the gap with traditional approaches.

Findings

PAM achieves up to 5x reduction in trainable parameters.

PAM outperforms state-of-the-art FM-based CIL methods.

PAM consistently mitigates catastrophic forgetting across benchmarks.

Abstract

The continual learning literature has rapidly shifted from traditional class incremental learning (CIL) techniques to foundation model (FM)-based CIL methods without a clear understanding of how these newer approaches compare to strong, lightweight convolutional baselines. This abrupt transition has created a substantial methodological gap, making it difficult to assess whether recent FM-based CIL progress reflects genuine advances or merely the absence of rigorous baselines. To address this gap, we introduce Pruned Adaptation Modules (PAM), a simple yet effective method that freezes the vast majority of the pre-trained ResNet while enabling scalable continual adaptation through sparse task-specific layers. PAM yields up to a ~5x reduction in trainable parameters and a ~6x reduction in total parameters, significantly reducing the cost of continual updates. Across diverse benchmarks, PAM consistently mitigates catastrophic forgetting and outperforms state-of-the-art FM-based CIL approaches. Our findings position PAM as a strong and transparent baseline that helps bridge the gap between traditional and FM-based CIL, guiding future research for a more accurate assessment of true progress in continual adaptation. The code can be found at: https://github.com/ElifCerenGokYildirim/PAM.