Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models

TL;DR

Ref-Long introduces a new benchmark to evaluate long-context referencing in language models, revealing significant challenges even for advanced models like GPT-4o, and providing insights through extensive analysis.

Contribution

This paper presents Ref-Long, a novel benchmark for assessing long-context referencing in LCLMs, highlighting existing shortcomings and offering comprehensive analysis.

Findings

Significant referencing shortcomings in 13 LCLMs, including GPT-4o.

Ref-Long's diverse scenarios reveal model limitations.

Analysis suggests need for improved long-context understanding.

Abstract

Long-context language models (LCLMs) have exhibited impressive capabilities in long-context understanding tasks. Among these, long-context referencing -- a crucial task that requires LCLMs to attribute items of interest to specific parts of long-context data -- remains underexplored. To bridge this gap, this paper proposes Referencing Evaluation for Long-context Language Models (Ref-Long), a novel benchmark designed to assess the long-context referencing capability of LCLMs. Specifically, Ref-Long requires LCLMs to identify the indexes of documents that reference a specific key, emphasizing contextual relationships between the key and the documents over simple retrieval. Based on the task design, we construct three subsets ranging from synthetic to realistic scenarios to form the Ref-Long benchmark. Experimental results of 13 LCLMs reveal significant shortcomings in long-context referencing, even among advanced models like GPT-4o. To further investigate these challenges, we conduct comprehensive analyses, including human evaluations, task format adjustments, fine-tuning experiments, and error analyses, leading to several key insights. Our data and code can be found in https://github. com/wujunjie1998/Ref-Long.