What is Wrong with Perplexity for Long-context Language Modeling?

TL;DR

This paper identifies the limitations of perplexity in evaluating long-context language models, introduces LongPPL as a better metric, and proposes LongCE for improved fine-tuning, leading to more accurate assessments and enhancements.

Contribution

The paper reveals why perplexity fails for long contexts, proposes LongPPL to focus on key tokens, and introduces LongCE for better model training.

Findings

LongPPL correlates strongly with long-context performance (Pearson -0.96)

Traditional PPL overlooks key tokens, misrepresenting model capabilities

LongCE improves model performance across benchmarks

Abstract

Handling long-context inputs is crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning. While recent approaches have extended the context windows of LLMs and employed perplexity (PPL) as a standard evaluation metric, PPL has proven unreliable for assessing long-context capabilities. The underlying cause of this limitation has remained unclear. In this work, we provide a comprehensive explanation for this issue. We find that PPL overlooks key tokens, which are essential for long-context understanding, by averaging across all tokens and thereby obscuring the true performance of models in long-context scenarios. To address this, we propose \textbf{LongPPL}, a novel metric that focuses on key tokens by employing a long-short context contrastive method to identify them. Our experiments demonstrate that LongPPL strongly correlates with performance on various long-context benchmarks (e.g., Pearson correlation of -0.96), significantly outperforming traditional PPL in predictive accuracy. Additionally, we introduce \textbf{LongCE} (Long-context Cross-Entropy) loss, a re-weighting strategy for fine-tuning that prioritizes key tokens, leading to consistent improvements across diverse benchmarks. In summary, these contributions offer deeper insights into the limitations of PPL and present effective solutions for accurately evaluating and enhancing the long-context capabilities of LLMs. Code is available at https://github.com/PKU-ML/LongPPL.