Marathon: A Race Through the Realm of Long Context with Large Language Models

TL;DR

Marathon introduces a new multiple-choice benchmark to accurately and efficiently evaluate large language models' comprehension and reasoning abilities over long contexts, addressing limitations of previous benchmarks.

Contribution

It presents a novel evaluation benchmark with a multiple-choice format that improves accuracy, speed, and fairness in assessing long-context understanding of LLMs.

Findings

Effective evaluation of LLMs' long-context comprehension

Benchmark outperforms traditional F1-based assessments

Supports assessment of optimization strategies for long-context generation

Abstract

With the advancement of large language models (LLMs) and the expansion of their context windows, existing long-context benchmarks fall short in effectively evaluating the models' comprehension and reasoning abilities in extended texts. Moreover, conventional benchmarks relying on F1 metrics often inaccurately score responses: they may undervalue correct answers that differ from the reference responses and overvalue incorrect ones that resemble the reference texts. In response to these limitations, we introduce Marathon, a novel evaluation benchmark that adopts a multiple-choice question format. It is specifically designed to overcome the constraints of previous benchmarks and provide a rapid, precise, and unbiased appraisal of the long-context comprehension skills of large language models. We conducted comprehensive evaluations on the Marathon benchmark with a range of state-of-the-art LLMs and assessed the effectiveness of various optimization strategies tailored for long-context generation. We anticipate that the Marathon benchmark and its associated leaderboard will enable a more precise and equitable evaluation of LLMs' capabilities in understanding and reasoning over extended contexts. Marathon is available at https://github.com/Hambaobao/Marathon.