Ability Transfer and Recovery via Modularized Parameters Localization

TL;DR

This paper investigates how abilities are distributed in large language models and introduces ACT, a method to localize and transfer ability-specific parameters, enabling recovery and integration of skills with minimal interference.

Contribution

The paper reveals that ability-related activations are concentrated in few channels and proposes ACT, a novel ability transfer method based on activation differences for modular parameter localization.

Findings

ACT effectively recovers forgotten abilities in multilingual reasoning tasks.

It enables merging multiple specialized models with minimal interference.

Ability-related activations are highly concentrated and disentangled in LLMs.

Abstract

Large language models can be continually pre-trained or fine-tuned to improve performance in specific domains, languages, or skills, but this specialization often degrades other capabilities and may cause catastrophic forgetting. We investigate how abilities are distributed within LLM parameters by analyzing module activations under domain- and language-specific inputs for closely related models. Across layers and modules, we find that ability-related activations are highly concentrated in a small set of channels (typically <5\%), and these channels are largely disentangled with good sufficiency and stability. Building on these observations, we propose ACT (Activation-Guided Channel-wise Ability Transfer), which localizes ability-relevant channels via activation differences and selectively transfers only the corresponding parameters, followed by lightweight fine-tuning for compatibility. Experiments on multilingual mathematical and scientific reasoning show that ACT can recover forgotten abilities while preserving retained skills. It can also merge multiple specialized models to integrate several abilities into a single model with minimal interference. Our code and data will be publicly released.