Data Difficulty and the Generalization--Extrapolation Tradeoff in LLM Fine-Tuning

TL;DR

This paper investigates how data difficulty influences large language model fine-tuning, revealing that the optimal difficulty level depends on dataset size and affects the generalization-extrapolation trade-off.

Contribution

It provides a systematic empirical and theoretical analysis showing the relationship between data difficulty, dataset size, and model generalization in fine-tuning LLMs.

Findings

Optimal data difficulty varies with dataset size.

Harder data becomes more beneficial as data budget increases.

The interplay between generalization gap and extrapolation gap explains the phenomenon.

Abstract

Data selection during supervised fine-tuning (SFT) can critically change the behavior of large language models (LLMs). Although existing work has studied the effect of selecting data based on heuristics such as perplexity, difficulty, or length, the reported findings are often inconsistent or context-dependent. In this work, we systematically study the role of data difficulty in fine-tuning from both empirical and theoretical perspectives, and find that there is no universally optimal difficulty level; rather, its effectiveness depends on the dataset size. We show that for a fixed data budget, there exists an optimal data difficulty for SFT, and that this optimal difficulty shifts toward harder data as the data budget increases. To explain this phenomenon, we conduct controlled synthetic experiments that reveal a simple underlying mechanism: the interplay between the (in-distribution) generalization gap and the extrapolation gap. We further support this mechanism through a theoretical analysis using PAC-Bayesian generalization bounds. Overall, our results clarify how data size and difficulty jointly affect the trade-off between generalization and extrapolation in SFT, providing guidance for difficulty-based data selection under certain model and data conditions.