Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning

TL;DR

This paper introduces Duct, a dual consolidation method that unifies and consolidates knowledge at representation and classifier levels to improve domain-incremental learning with pre-trained models, reducing catastrophic forgetting.

Contribution

The paper proposes a novel dual consolidation approach that merges representations and classifiers to retain knowledge across multiple domains in incremental learning.

Findings

Achieves state-of-the-art results on four benchmark datasets.

Effectively consolidates knowledge to prevent catastrophic forgetting.

Demonstrates robustness across diverse domain shifts.

Abstract

Domain-Incremental Learning (DIL) involves the progressive adaptation of a model to new concepts across different domains. While recent advances in pre-trained models provide a solid foundation for DIL, learning new concepts often results in the catastrophic forgetting of pre-trained knowledge. Specifically, sequential model updates can overwrite both the representation and the classifier with knowledge from the latest domain. Thus, it is crucial to develop a representation and corresponding classifier that accommodate all seen domains throughout the learning process. To this end, we propose DUal ConsolidaTion (Duct) to unify and consolidate historical knowledge at both the representation and classifier levels. By merging the backbone of different stages, we create a representation space suitable for multiple domains incrementally. The merged representation serves as a balanced intermediary that captures task-specific features from all seen domains. Additionally, to address the mismatch between consolidated embeddings and the classifier, we introduce an extra classifier consolidation process. Leveraging class-wise semantic information, we estimate the classifier weights of old domains within the latest embedding space. By merging historical and estimated classifiers, we align them with the consolidated embedding space, facilitating incremental classification. Extensive experimental results on four benchmark datasets demonstrate Duct's state-of-the-art performance. Code is available at https://github.com/Estrella-fugaz/CVPR25-Duct