Performance Law of Large Language Models

TL;DR

This paper introduces an empirical 'Performance Law' equation that accurately predicts the MMLU score of large language models based on key hyperparameters and training data size, aiding practical development and resource allocation.

Contribution

The paper presents a novel empirical equation to directly estimate LLM performance, moving beyond loss-based scaling laws for more practical predictions.

Findings

Accurately predicts MMLU scores across diverse LLMs

Uses only a few hyperparameters and data size for predictions

Guides architecture choice and resource allocation

Abstract

Guided by the belief of the scaling law, large language models (LLMs) have achieved impressive performance in recent years. However, scaling law only gives a qualitative estimation of loss, which is influenced by various factors such as model architectures, data distributions, tokenizers, and computation precision. Thus, estimating the real performance of LLMs with different training settings rather than loss may be quite useful in practical development. In this article, we present an empirical equation named "Performance Law" to directly predict the MMLU score of an LLM, which is a widely used metric to indicate the general capability of LLMs in real-world conversations and applications. Based on only a few key hyperparameters of the LLM architecture and the size of training data, we obtain a quite accurate MMLU prediction of various LLMs with diverse sizes and architectures developed by different organizations in different years. Performance law can be used to guide the choice of LLM architecture and the effective allocation of computational resources without extensive experiments.