Emergent Abilities of Large Language Models

TL;DR

This paper explores the unpredictable emergence of new abilities in large language models as they scale, which are not present in smaller models and cannot be predicted by performance extrapolation.

Contribution

It introduces the concept of emergent abilities in large language models, highlighting their unpredictable nature and implications for future scaling.

Findings

Emergent abilities appear only in larger models.

Emergence cannot be predicted from smaller models' performance.

Scaling can further expand language model capabilities.

Abstract

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.