Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need

TL;DR

This paper introduces a method to predict emergent abilities of large language models using proxy tasks, enabling better anticipation of capabilities without extensive training on large models.

Contribution

The authors propose a novel approach that leverages proxy tasks and relevance metrics to predict emergent abilities of LLMs, addressing limitations of scaling laws.

Findings

Strong correlation between predicted and actual performance in tool utilization tasks

Method effectively identifies relevant proxy tasks for emergent abilities

Predictive accuracy improves with ensemble validation

Abstract

While scaling laws optimize training configurations for large language models (LLMs) through experiments on smaller or early-stage models, they fail to predict emergent abilities due to the absence of such capabilities in these models. To address this, we propose a method that predicts emergent abilities by leveraging proxy tasks. We begin by establishing relevance metrics between the target task and candidate tasks based on performance differences across multiple models. These candidate tasks are then validated for robustness with small model ensembles, leading to the selection of the most appropriate proxy tasks. The predicted performance on the target task is then derived by integrating the evaluation results of these proxies. In a case study on tool utilization capabilities, our method demonstrated a strong correlation between predicted and actual performance, confirming its effectiveness.