Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning

TL;DR

Sparse Memory Finetuning (SMF) enhances task adaptation of pretrained language models while minimizing catastrophic forgetting by updating only a small subset of memory rows, outperforming LoRA and full finetuning on specific metrics.

Contribution

The paper re-implements SMF on Qwen-2.5-0.5B-Instruct and compares its effectiveness against LoRA and full finetuning, highlighting its ability to reduce forgetting while maintaining task performance.

Findings

SMF improves MedMCQA accuracy by 2.5 percentage points.

SMF maintains forgetting probes within 1 point of the base model.

Different row-selection rules (KL-divergence and TF-IDF) balance forgetting metrics differently.

Abstract

Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting. Sparse Memory Finetuning (SMF) tries to avoid this by adding key-value memory layers to the model and, on each training step, updating only the small set of memory rows that the current batch reads most heavily. We re-implement SMF on Qwen-2.5-0.5B-Instruct and compare it with LoRA and full finetuning on MedMCQA, a 4-choice medical exam task, using WikiText perplexity and TriviaQA accuracy as forgetting probes. SMF improves MedMCQA by 2.5 percentage points while keeping both forgetting probes within roughly 1 point of the base model, whereas LoRA and full finetuning achieve larger gains but with clear drift on both. We also compare two row-selection rules (KL-divergence and TF-IDF), which balance the two forgetting metrics differently.