Representation in large language models

TL;DR

This paper investigates whether large language models operate through representation-based processing or mere memorization, arguing for the former and proposing methods to analyze their internal representations.

Contribution

It advances the debate on LLM mechanisms by defending the view that they utilize representations and offers practical techniques for studying these representations.

Findings

LLMs are partially driven by representation-based information processing.

Proposes techniques to analyze and interpret LLM internal representations.

Provides a foundation for future theoretical work on language models.

Abstract

The extraordinary success of recent Large Language Models (LLMs) on a diverse array of tasks has led to an explosion of scientific and philosophical theorizing aimed at explaining how they do what they do. Unfortunately, disagreement over fundamental theoretical issues has led to stalemate, with entrenched camps of LLM optimists and pessimists often committed to very different views of how these systems work. Overcoming stalemate requires agreement on fundamental questions, and the goal of this paper is to address one such question, namely: is LLM behavior driven partly by representation-based information processing of the sort implicated in biological cognition, or is it driven entirely by processes of memorization and stochastic table look-up? This is a question about what kind of algorithm LLMs implement, and the answer carries serious implications for higher level questions about whether these systems have beliefs, intentions, concepts, knowledge, and understanding. I argue that LLM behavior is partially driven by representation-based information processing, and then I describe and defend a series of practical techniques for investigating these representations and developing explanations on their basis. The resulting account provides a groundwork for future theorizing about language models and their successors.