Residual SODAP: Residual Self-Organizing Domain-Adaptive Prompting with Structural Knowledge Preservation for Continual Learning

TL;DR

Residual SODAP introduces a novel continual learning framework that combines prompt adaptation and classifier preservation, effectively mitigating catastrophic forgetting in domain-incremental scenarios without task IDs or extra data.

Contribution

It proposes a joint prompt-based and classifier-level approach with residual aggregation, sparse prompt selection, and data-free distillation for improved continual learning.

Findings

Achieves state-of-the-art accuracy on three DIL benchmarks.

Effectively mitigates catastrophic forgetting without task IDs.

Demonstrates robustness across different domain shifts.

Abstract

Continual learning (CL) suffers from catastrophic forgetting, which is exacerbated in domain-incremental learning (DIL) where task identifiers are unavailable and storing past data is infeasible. While prompt-based CL (PCL) adapts representations with a frozen backbone, we observe that prompt-only improvements are often insufficient due to suboptimal prompt selection and classifier-level instability under domain shifts. We propose Residual SODAP, which jointly performs prompt-based representation adaptation and classifier-level knowledge preservation. Our framework combines $\alpha$-entmax sparse prompt selection with residual aggregation, data-free distillation with pseudo-feature replay, prompt-usage--based drift detection, and uncertainty-aware multi-loss balancing. Across three DIL benchmarks without task IDs or extra data storage, Residual SODAP achieves state-of-the-art AvgACC/AvgF of 0.850/0.047 (DR), 0.760/0.031 (Skin Cancer), and 0.995/0.003 (CORe50).