Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization

TL;DR

Easy2Hard-Bench introduces a standardized collection of benchmark datasets with difficulty labels across various domains, enabling systematic profiling of LLM performance and generalization from easy to hard problems.

Contribution

The paper presents a new benchmark collection with fine-grained difficulty annotations and applies established ranking systems to evaluate LLMs across difficulty levels.

Findings

State-of-the-art LLMs show varied performance across difficulty levels.

Datasets contain a higher proportion of challenging problems than previous collections.

Analysis reveals insights into LLM generalization capabilities.

Abstract

While generalization over tasks from easy to hard is crucial to profile language models (LLMs), the datasets with fine-grained difficulty annotations for each problem across a broad range of complexity are still blank. Aiming to address this limitation, we present Easy2Hard-Bench, a consistently formatted collection of 6 benchmark datasets spanning various domains, such as mathematics and programming problems, chess puzzles, and reasoning questions. Each problem within these datasets is annotated with numerical difficulty scores. To systematically estimate problem difficulties, we collect abundant performance data on attempts to each problem by humans in the real world or LLMs on the prominent leaderboard. Leveraging the rich performance data, we apply well-established difficulty ranking systems, such as Item Response Theory (IRT) and Glicko-2 models, to uniformly assign numerical difficulty scores to problems. Moreover, datasets in Easy2Hard-Bench distinguish themselves from previous collections by a higher proportion of challenging problems. Through extensive experiments with six state-of-the-art LLMs, we provide a comprehensive analysis of their performance and generalization capabilities across varying levels of difficulty, with the aim of inspiring future research in LLM generalization. The datasets are available at https://huggingface.co/datasets/furonghuang-lab/Easy2Hard-Bench.