Attribution-Guided Continual Learning for Large Language Models

TL;DR

This paper introduces an attribution-guided fine-tuning method for LLMs that estimates parameter importance to mitigate catastrophic forgetting in continual learning.

Contribution

It proposes a novel framework that uses semantic attribution to identify and preserve important parameters for previous tasks during continual learning.

Findings

Outperforms baseline methods on continual learning benchmarks.

Achieves better retention of old tasks without sacrificing new task performance.

Effectively modulates parameter updates based on task-specific importance.

Abstract

Large language models (LLMs) often suffer from catastrophic forgetting in continual learning: after learning new tasks sequentially, they perform worse on earlier tasks. Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization. However, these methods lack semantic awareness of internal knowledge distribution in LLMs. As a result, they cannot distinguish parameters that should be preserved or updated. We propose an attribution-guided continual fine-tuning framework for LLMs. Our method estimates task-specific, element-wise parameter importance in each Transformer layer and uses these scores to modulate gradients. Parameters important to previous tasks receive smaller updates, while less relevant ones remain plastic for learning new tasks. Experiments on continual learning benchmarks show that our method consistently outperforms baselines, achieving better retention of old tasks while maintaining competitive performance on new tasks.