MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models

TL;DR

MiniLongBench is a compact, low-cost benchmark for evaluating long context understanding in large language models, reducing evaluation costs significantly while maintaining high correlation with existing benchmarks.

Contribution

We introduce MiniLongBench, a concise and efficient benchmark created by pruning existing data, enabling cost-effective evaluation of LLMs' long context understanding capabilities.

Findings

Evaluation cost reduced to 4.5% of original

High rank correlation coefficient of 0.97 with LongBench

MiniLongBench covers six major task categories

Abstract

Long Context Understanding (LCU) is a critical area for exploration in current large language models (LLMs). However, due to the inherently lengthy nature of long-text data, existing LCU benchmarks for LLMs often result in prohibitively high evaluation costs, like testing time and inference expenses. Through extensive experimentation, we discover that existing LCU benchmarks exhibit significant redundancy, which means the inefficiency in evaluation. In this paper, we propose a concise data compression method tailored for long-text data with sparse information characteristics. By pruning the well-known LCU benchmark LongBench, we create MiniLongBench. This benchmark includes only 237 test samples across six major task categories and 21 distinct tasks. Through empirical analysis of over 60 LLMs, MiniLongBench achieves an average evaluation cost reduced to only 4.5% of the original while maintaining an average rank correlation coefficient of 0.97 with LongBench results. Therefore, our MiniLongBench, as a low-cost benchmark, holds great potential to substantially drive future research into the LCU capabilities of LLMs. See https://github.com/MilkThink-Lab/MiniLongBench for our code, data and tutorial.