Establishing Task Scaling Laws via Compute-Efficient Model Ladders

TL;DR

This paper introduces task scaling laws and model ladders to efficiently predict the performance of large pretrained language models on various tasks, reducing the need for extensive training.

Contribution

It presents a novel two-step prediction method using small-scale ladder models to accurately forecast large model task performance with minimal compute.

Findings

Predicted target model accuracy within 2 points of actual results.

Higher prediction errors correlate with greater metric variance.

Training ladder models costs only 1% of target model compute.

Abstract

We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting. Standard power laws for language modeling loss cannot accurately model task performance. Therefore, we leverage a two-step prediction approach: (1) use model and data size to predict an intermediate loss, then (2) use it to predict task performance. We train a set of small-scale "ladder" models, collect data points to fit the parameterized functions of the two prediction steps, and make predictions for two target models: a 7B model trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder models only costs 1% of the compute used for the target models. On four multiple-choice tasks formatted as ranked classification, we can predict the accuracy of both target models within 2 points of absolute error. We find that tasks with higher prediction error also have higher variance in the metrics over model checkpoints. We also contrast multiple design choices for predicting accuracy, and present recommendations for extending our method to new models and tasks.