Unveiling Over-Memorization in Finetuning LLMs for Reasoning Tasks

TL;DR

This paper investigates the over-memorization phenomenon in finetuned large language models for reasoning tasks, revealing its prevalence, effects on robustness, and proposing mitigation strategies.

Contribution

It uncovers the over-memorization phenomenon during LLM finetuning, analyzes its causes, and offers techniques to mitigate its negative effects.

Findings

Over-memorization occurs during specific finetuning stages.

Over-memorized models have reduced robustness and generalization.

Proposed techniques help mitigate over-memorization effects.

Abstract

The pretrained large language models (LLMs) are finetuned with labeled data for better instruction following ability and alignment with human values. In this paper, we study the learning dynamics of LLM finetuning on reasoning tasks and reveal the uncovered over-memorization phenomenon during a specific stage of LLM finetuning. At this stage, the LLMs have excessively memorized training data and exhibit high test perplexity while maintaining good test accuracy. We explore the conditions that contribute to over-memorization and discover that this issue is prevalent across various tasks, models, and fine-tuning methods, with prolonged training and large learning rates exacerbating the problem. Although models with over-memorization demonstrate comparable test accuracy to normal models, they suffer from reduced robustness, poor out-of-distribution generalization, and decreased generation diversity. In light of our findings on over-memorization, we offer recommendations for checkpoint selection and propose techniques such as checkpoint merging and memorization-aware reweighting to mitigate this effect.