$\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens

TL;DR

This paper introduces $ abla$Bench, a new benchmark with over 100K tokens to evaluate LLMs' ability to process extremely long contexts, revealing current models' limitations and guiding future improvements.

Contribution

It presents the first long-context benchmark exceeding 100K tokens, including diverse tasks in English and Chinese, to evaluate and compare LLMs' long dependency understanding.

Findings

Existing long-context LLMs need significant improvements.

Current models struggle with 100K+ token contexts.

Analysis reveals specific behaviors in LLMs processing long contexts.

Abstract

Processing and reasoning over long contexts is crucial for many practical applications of Large Language Models (LLMs), such as document comprehension and agent construction. Despite recent strides in making LLMs process contexts with more than 100K tokens, there is currently a lack of a standardized benchmark to evaluate this long-context capability. Existing public benchmarks typically focus on contexts around 10K tokens, limiting the assessment and comparison of LLMs in processing longer contexts. In this paper, we propose $\infty$Bench, the first LLM benchmark featuring an average data length surpassing 100K tokens. $\infty$Bench comprises synthetic and realistic tasks spanning diverse domains, presented in both English and Chinese. The tasks in $\infty$Bench are designed to require well understanding of long dependencies in contexts, and make simply retrieving a limited number of passages from contexts not sufficient for these tasks. In our experiments, based on $\infty$Bench, we evaluate the state-of-the-art proprietary and open-source LLMs tailored for processing long contexts. The results indicate that existing long context LLMs still require significant advancements to effectively process 100K+ context. We further present three intriguing analyses regarding the behavior of LLMs processing long context.