Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Model

TL;DR

This paper introduces a solvable linear model that links neural scaling laws to inference, showing how training reduces the impact of hard-to-learn instances and predicting the behavior of pass@$k$ failure rates.

Contribution

It presents the Latent Instance Difficulty (LID) model, connecting training-dependent inference scaling to the tail of target difficulty distributions, with testable predictions.

Findings

Pass@$k$ failure rate follows a power-law decay with training-dependent exponent.

Training shrinks the hard tail of the error distribution, improving generalization.

Model predictions validated on simulations and real data proxies.

Abstract

We analyze neural scaling laws in a solvable model of last-layer fine-tuning where targets have intrinsic, instance-heterogeneous difficulty. In our Latent Instance Difficulty (LID) model, each input's target variance is governed by a latent ``precision'' drawn from a heavy-tailed distribution. While generalization loss recovers standard scaling laws, our main contribution connects this to inference. The pass@$k$ failure rate exhibits a power-law decay, $k^{-\beta_\text{eff}}$, but the observed exponent $\beta_\text{eff}$ is training-dependent. It grows with sample size $N$ before saturating at an intrinsic limit $\beta$ set by the difficulty distribution's tail. This coupling reveals that learning shrinks the ``hard tail'' of the error distribution: improvements in the model's generalization error steepen the pass@$k$ curve until irreducible target variance dominates. The LID model yields testable, closed-form predictions for this behavior, including a compute-allocation rule that favors training before saturation and inference attempts after. We validate these predictions in simulations and in two real-data proxies: CIFAR-10H (human-label variance) and a maths teacher-student distillation task.