Stable Routing for Mixture-of-Experts in Class-Incremental Learning

TL;DR

This paper introduces StaR-MoE, a routing-level framework that stabilizes expert routing in mixture-of-experts models for class-incremental learning, improving knowledge retention and adaptation.

Contribution

It proposes sensitivity-aware routing alignment and asymmetric capacity regularization to enhance expert stability and capacity utilization in CIL.

Findings

StaR-MoE outperforms state-of-the-art methods on four CIL benchmarks.

Stable routing significantly improves both average and last accuracy.

The framework effectively balances old-class knowledge preservation and new-class learning.

Abstract

Class-incremental learning (CIL) requires models to learn new classes sequentially while preserving prior knowledge. Recently, approaches that combine pre-trained models with mixture-of-experts (MoE) have received increasing attention in CIL: they typically expand experts during learning and employ a router to assign weights across experts. However, existing MoE methods often overlook routing drift induced by expert expansion. Once new experts are introduced, the router may reassign samples from earlier classes to newly added experts, thereby perturbing previously established expert compositions and causing interference even when old experts remain frozen. We argue that expandable MoE in CIL requires two complementary properties: stable old-class routing for knowledge preservation and sufficient capacity utilization for new-class adaptation. To this end, we propose Stable Routing for MoE (StaR-MoE), a routing-level framework for expandable MoE in CIL. By incorporating sensitivity-aware routing alignment, StaR-MoE aligns current old-class routing behavior with historical routing distributions through sensitivity-guided constraints. Complementarily, StaR-MoE introduces asymmetric capacity regularization to encourage effective utilization of the expanded expert pool without compromising class-specific routing specialization. Extensive experiments across four standard CIL benchmarks demonstrate that StaR-MoE consistently improves both average and last accuracy over state-of-the-art methods, highlighting the importance of stable routing.