Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve

TL;DR

This paper proposes a teleological approach to understanding large language models by analyzing the problem they are trained to solve, revealing how probabilities influence their accuracy and failure modes.

Contribution

It introduces the probabilistic factors influencing LLM performance and empirically tests these predictions on GPT-3.5 and GPT-4 across multiple tasks.

Findings

LLMs perform better when task and output probabilities are high

GPT-4's cipher decoding accuracy drops from 51% to 13% in low-probability scenarios

LLMs are shaped by their training pressures, not human-like cognition

Abstract

The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that in order to develop a holistic understanding of these systems we need to consider the problem that they were trained to solve: next-word prediction over Internet text. By recognizing the pressures that this task exerts we can make predictions about the strategies that LLMs will adopt, allowing us to reason about when they will succeed or fail. This approach - which we call the teleological approach - leads us to identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input. We predict that LLMs will achieve higher accuracy when these probabilities are high than when they are low - even in deterministic settings where probability should not matter. To test our predictions, we evaluate two LLMs (GPT-3.5 and GPT-4) on eleven tasks, and we find robust evidence that LLMs are influenced by probability in the ways that we have hypothesized. In many cases, the experiments reveal surprising failure modes. For instance, GPT-4's accuracy at decoding a simple cipher is 51% when the output is a high-probability word sequence but only 13% when it is low-probability. These results show that AI practitioners should be careful about using LLMs in low-probability situations. More broadly, we conclude that we should not evaluate LLMs as if they are humans but should instead treat them as a distinct type of system - one that has been shaped by its own particular set of pressures.