Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution

TL;DR

MADE-IT is an adaptive continual model merging method that uses manifold geometry to manage experts efficiently, avoiding redundancy and improving performance without retraining.

Contribution

It introduces a manifold-aware expert evolution and implicit routing mechanism that operates without training or parameterized gating.

Findings

Outperforms baselines in accuracy and robustness across task sequences.

Effectively prunes redundant experts, especially in early layers.

Demonstrates superior long-term continual learning performance.

Abstract

Continual Model Merging (CMM) sequentially integrates task-specific models into a unified architecture without intensive retraining. However, existing CMM methods are hindered by a fundamental saturation-redundancy dilemma: backbone-centric approaches face parameter saturation and representation interference within fixed capacities, whereas Mixture-of-Experts (MoE) variants resort to indiscriminate expansion, incurring expert redundancy and a routing bottleneck reliant on additional data-driven optimization. To resolve these challenges, we propose MADE-IT (Manifold-Aware Dynamic Expert Evolution and Implicit rouTing), an adaptive CMM method that orchestrates expert management and activation by grounding intrinsic expert representations in manifold geometry. We introduce a projection-based subspace affinity metric coupled with a distribution-aware adaptive threshold mechanism to guide autonomous expert evolution, harmonizing diversity with architectural parsimony. Furthermore, to bypass parameterized gating networks, we design a data-free and training-free implicit routing mechanism that activates experts via feature-subspace alignment. Extensive experiments demonstrate that MADE-IT consistently outperforms strong baselines in accuracy and robustness across long-horizon and shuffled task sequences, while significantly pruning redundant experts, particularly within generic modules and early layers.