CKAA: Cross-subspace Knowledge Alignment and Aggregation for Robust Continual Learning

TL;DR

CKAA introduces a novel framework for continual learning that aligns feature subspaces and adaptively aggregates knowledge, significantly improving robustness against misleading task-ids in parameter-efficient fine-tuning methods.

Contribution

The paper proposes CKAA, a new framework with dual-level knowledge alignment and task-confidence-guided aggregation, enhancing robustness and accuracy in continual learning.

Findings

CKAA outperforms existing PEFT-based CL methods in experiments.

Dual-level knowledge alignment improves feature discrimination across subspaces.

Task-confidence-guided aggregation reduces errors caused by misleading task-ids.

Abstract

Continual Learning (CL) empowers AI models to continuously learn from sequential task streams. Recently, parameter-efficient fine-tuning (PEFT)-based CL methods have garnered increasing attention due to their superior performance. They typically allocate a unique sub-module for learning each task, with a task recognizer to select the appropriate sub-modules for testing images. However, due to the feature subspace misalignment from independently trained sub-modules, these methods tend to produce ambiguous decisions under misleading task-ids. To address this, we propose Cross-subspace Knowledge Alignment and Aggregation (CKAA), a novel framework that enhances model robustness against misleading task-ids through two key innovations: (1) Dual-level Knowledge Alignment (DKA): By aligning intra-class feature distributions across different subspaces and learning a robust global classifier through a feature simulation process, DKA enables the model to distinguish features from both correct and incorrect subspaces during training. (2) Task-Confidence-guided Mixture of Adapters (TC-MoA): A robust inference scheme that adaptively aggregates task-specific knowledge from relevant sub-modules based on task-confidence scores, avoiding overconfidence in misleading task-id predictions. Extensive experiments demonstrate that CKAA outperforms existing PEFT-based CL methods.