From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning

TL;DR

This paper introduces SAEF, a hierarchical, semantic-guided expert forest for class-incremental learning that improves knowledge sharing and achieves state-of-the-art results by organizing adapters into clusters and trees.

Contribution

SAEF is the first method to structure adapters hierarchically based on semantic task relationships for improved class-incremental learning.

Findings

SAEF outperforms existing methods on benchmark datasets.

Hierarchical organization enhances knowledge sharing among tasks.

Adaptive expert selection improves prediction accuracy.

Abstract

Class-Incremental Learning (CIL) requires models to learn new classes without forgetting old ones. A common method is to freeze a pre-trained model and train a new, lightweight adapter for each task. While this prevents forgetting, it treats the learned knowledge as a simple, unstructured collection and fails to use the relationships between tasks. To this end, we propose the Semantic-guided Adaptive Expert Forest (SAEF), a new method that organizes adapters into a structured hierarchy for better knowledge sharing. SAEF first groups tasks into conceptual clusters based on their semantic relationships. Then, within each cluster, it builds a balanced expert tree by creating new adapters from merging the adapters of similar tasks. At inference time, SAEF finds and activates a set of relevant experts from the forest for any given input. The final prediction is made by combining the outputs of these activated experts, weighted by how confident each expert is. Experiments on several benchmark datasets show that SAEF achieves SOTA performance.