How do Language Models Bind Entities in Context?

TL;DR

This paper uncovers a binding ID mechanism in large language models that enables them to associate entities with attributes in context, revealing how they internally represent symbolic knowledge for in-context reasoning.

Contribution

It identifies and characterizes a binding ID mechanism in large language models, demonstrating its role in representing entity-attribute associations internally.

Findings

Binding ID vectors are present in all large models studied.

Binding ID vectors form a continuous, discernable subspace.

Interventions show binding IDs are crucial for in-context reasoning.

Abstract

To correctly use in-context information, language models (LMs) must bind entities to their attributes. For example, given a context describing a "green square" and a "blue circle", LMs must bind the shapes to their respective colors. We analyze LM representations and identify the binding ID mechanism: a general mechanism for solving the binding problem, which we observe in every sufficiently large model from the Pythia and LLaMA families. Using causal interventions, we show that LMs' internal activations represent binding information by attaching binding ID vectors to corresponding entities and attributes. We further show that binding ID vectors form a continuous subspace, in which distances between binding ID vectors reflect their discernability. Overall, our results uncover interpretable strategies in LMs for representing symbolic knowledge in-context, providing a step towards understanding general in-context reasoning in large-scale LMs.