LongGenBench: Long-context Generation Benchmark

TL;DR

LongGenBench is a new benchmark designed to evaluate the ability of large language models to generate coherent, long-context responses, revealing performance degradation across different models and configurations.

Contribution

This paper introduces LongGenBench, a synthetic benchmark specifically for assessing long-context generation capabilities of LLMs, filling a gap in existing evaluation tools.

Findings

Models show 1.2% to 47.1% performance degradation in long-context generation.

Gemini-1.5-Flash exhibits the least degradation among API models.

Qwen2 series shows the least degradation among open source models.

Abstract

Current long-context benchmarks primarily focus on retrieval-based tests, requiring Large Language Models (LLMs) to locate specific information within extensive input contexts, such as the needle-in-a-haystack (NIAH) benchmark. Long-context generation refers to the ability of a language model to generate coherent and contextually accurate text that spans across lengthy passages or documents. While recent studies show strong performance on NIAH and other retrieval-based long-context benchmarks, there is a significant lack of benchmarks for evaluating long-context generation capabilities. To bridge this gap and offer a comprehensive assessment, we introduce a synthetic benchmark, LongGenBench, which allows for flexible configurations of customized generation context lengths. LongGenBench advances beyond traditional benchmarks by redesigning the format of questions and necessitating that LLMs respond with a single, cohesive long-context answer. Upon extensive evaluation using LongGenBench, we observe that: (1) both API accessed and open source models exhibit performance degradation in long-context generation scenarios, ranging from 1.2% to 47.1%; (2) different series of LLMs exhibit varying trends of performance degradation, with the Gemini-1.5-Flash model showing the least degradation among API accessed models, and the Qwen2 series exhibiting the least degradation in LongGenBench among open source models.