Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning

TL;DR

This paper investigates how reducing high perplexity tokens in LLM-generated data during fine-tuning improves cross-domain robustness and mitigates forgetting, providing empirical insights into token-level effects on model performance.

Contribution

It introduces a novel explanation that lowering high perplexity tokens in training data helps preserve non-target task performance during fine-tuning.

Findings

Reducing high perplexity tokens improves non-target task robustness.

Masking high perplexity tokens in ground truth data yields similar benefits.

Empirical validation across multiple models supports the token perplexity hypothesis.

Abstract

Maintaining consistent model performance across domains is a fundamental challenge in machine learning. While recent work has explored using LLM-generated data for fine-tuning, its impact on cross-domain generalization remains poorly understood. This paper presents a systematic analysis revealing that fine-tuning with LLM-generated data not only improves target task performance but also reduces non-target task degradation compared to fine-tuning with ground truth data. Through analyzing the data sequence in tasks of various domains, we demonstrate that this enhancement of non-target task robustness stems from the reduction of high perplexity tokens found in LLM-generated sequences. Following our findings, we showed that masking high perplexity tokens in ground truth training data achieves similar non-target task performance preservation, comparable to using LLM-generated data. Extensive experiments across different model families and scales, including Gemma 2 IT 2B, Llama 3 8B Instruct, and three additional models, agree with our findings. To the best of our knowledge, this is the first work to provide an empirical explanation based on token perplexity reduction to mitigate catastrophic forgetting in LLMs after fine-tuning, offering valuable insights for developing more robust fine-tuning strategies.