Dynamical Adapter Fusion: Constructing A Global Adapter for Pre-Trained Model-based Class-Incremental Learning

TL;DR

This paper introduces Dynamical Adapter Fusion, a method to create a single, robust global adapter for class-incremental learning from pre-trained models, improving knowledge transfer and reducing forgetting.

Contribution

The paper proposes a novel fusion mechanism based on PAC-Bayes theory and Taylor expansion to dynamically combine task-specific adapters into one global adapter.

Findings

Achieves state-of-the-art performance on multiple CIL benchmarks.

Effectively balances stability and plasticity in continual learning.

Reduces retrieval costs and interference compared to naive fusion methods.

Abstract

Class-Incremental Learning (CIL) requires models to continuously acquire new classes without forgetting previously learned ones. A dominant paradigm involves freezing a pre-trained model and training lightweight, task-specific adapters. However, maintaining task-specific parameters hinders knowledge transfer and incurs high retrieval costs, while naive parameter fusion often leads to destructive interference and catastrophic forgetting. To address these challenges, we propose Dynamical Adapter Fusion (DAF) to construct a single robust global adapter. Grounded in the PAC-Bayes theorem, we derive a fusion mechanism that explicitly integrates three components: the optimized task-specific adapter parameters, the previous global adapter parameters, and the initialization parameters. We utilize the Taylor expansion of the loss function to derive the optimal fusion coefficients, dynamically achieving the best balance between stability and plasticity. Furthermore, we propose a Robust Initialization strategy to effectively capture global knowledge patterns. Experiments on multiple CIL benchmarks demonstrate that DAF achieves state-of-the-art (SOTA) performance.