Demystifying Language Model Forgetting with Low-rank Example Associations

TL;DR

This paper investigates how fine-tuning large language models causes forgetting of upstream knowledge, revealing that such forgetting can be modeled with low-rank matrices, enabling efficient prediction and mitigation of forgotten examples.

Contribution

It introduces a low-rank matrix approximation approach to analyze and predict forgetting in LLMs, outperforming prior semantic-based methods and enabling targeted mitigation.

Findings

Low-rank matrices effectively model forgetting patterns.

Matrix completion accurately predicts forgotten examples.

Upweighting predicted examples reduces forgetting during fine-tuning.

Abstract

Large language models (LLMs) suffer from forgetting of upstream knowledge when fine-tuned. Despite efforts on mitigating forgetting, few have investigated how forgotten upstream examples are dependent on newly learned tasks. Insights on such dependencies enable efficient and targeted mitigation of forgetting. In this paper, we empirically analyze forgetting that occurs in $N$ upstream examples of language modeling or instruction-tuning after fine-tuning LLMs on one of $M$ new tasks, visualized in $M\times N$ matrices. We show that the matrices are often well-approximated with low-rank matrices, indicating the dominance of simple associations between the learned tasks and forgotten upstream examples. Leveraging the analysis, we predict forgetting of upstream examples when fine-tuning LLMs on unseen tasks with matrix completion over the empirical associations. This enables fast identification of most forgotten examples without expensive inference on the entire upstream data. Despite simplicity, the approach outperforms prior approaches that learn semantic relationships of learned tasks and upstream examples with LMs. We demonstrate the practical utility of our analysis by showing statistically significantly reduced forgetting as we upweight predicted examples for replay during fine-tuning. Code, data, and statistics collected: https://github.com/AuCson/low-rank-forgetting