LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges?

TL;DR

This paper introduces LooGLE v2, a benchmark for evaluating large language models' ability to understand and process long dependencies in real-world scenarios, revealing significant limitations despite extended context windows.

Contribution

The paper presents a new benchmark with real-world long texts and diverse tasks to assess LLMs' long-context understanding, highlighting their current limitations.

Findings

Best models score only 59.2% on the benchmark

Popular LLMs understand much shorter contexts than claimed

Significant room for improvement in long-context understanding

Abstract

Large language models (LLMs) are equipped with increasingly extended context windows recently, yet their long context understanding capabilities over long dependency tasks remain fundamentally limited and underexplored. This gap is especially significant in many real-world long-context applications that were rarely benchmarked. In this paper, we introduce LooGLE v2, a novel benchmark designed to evaluate LLMs' long context ability in real-world applications and scenarios. Our benchmark consists of automatically collected real-world long texts, ranging from 16k to 2M tokens, encompassing domains in law, finance, game and code. Accordingly, we delicately design 10 types of domain-specific long-dependency tasks and generate 1,934 QA instances with various diversity and complexity in a scalable data curation pipeline for further practical needs. We conduct a comprehensive assessment of 6 locally deployed and 4 API-based LLMs. The evaluation results show that even the best-performing model achieves only a 59.2% overall score on our benchmark. Despite the extensive context windows, popular LLMs are only capable of understanding a much shorter length of context than they claim to be, revealing significant limitations in their ability to handle real-world tasks with long dependencies and highlighting substantial room for model improvement in practical long-context understanding.