Revisiting Generalization Across Difficulty Levels: It's Not So Easy

TL;DR

This paper systematically evaluates how large language models generalize across different task difficulties, revealing that training on easy or hard data alone often does not lead to consistent improvements, emphasizing the need for diverse difficulty levels.

Contribution

It introduces a novel, objective difficulty ranking method for datasets based on multiple LLMs and IRT, providing a finer-grained analysis of LLM generalization across difficulty levels.

Findings

Cross-difficulty generalization is often limited.

Training on easy or hard data alone does not guarantee improvements.

Diverse difficulty data is crucial for effective LLM training and evaluation.

Abstract

We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation. Existing research is mixed regarding whether training on easier or harder data leads to better results, and whether those gains come on easier or harder test data. We address this question by conducting a systematic evaluation of LLMs' generalization across models, datasets, and fine-grained groups of example difficulty. We rank examples in six datasets using the outputs of thousands of different LLMs and Item Response Theory (IRT), a well-established difficulty metric in educational testing. Unlike prior work, our difficulty ratings are therefore determined solely by the abilities of many different LLMs, excluding human opinions of difficulty. With a more objective, larger-scale, and finer-grained analysis, we show that cross-difficulty generalization is often limited; training on either easy or hard data cannot achieve consistent improvements across the full range of difficulties. These results show the importance of having a range of difficulties in both training and evaluation data for LLMs, and that taking shortcuts with respect to difficulty is risky.