Emergent LLM behaviors are observationally equivalent to data leakage

TL;DR

This paper argues that behaviors observed in large language models during a naming game are better explained by data leakage and memorization of training data rather than emergent social conventions.

Contribution

The study demonstrates that what appears as emergent social behaviors in LLMs can be attributed to data leakage and memorization, challenging previous interpretations.

Findings

Models recognize the structure of the coordination game.

Models recall outcomes from pre-training data.

Observed behaviors are indistinguishable from memorization.

Abstract

Ashery et al. recently argue that large language models (LLMs), when paired to play a classic "naming game," spontaneously develop linguistic conventions reminiscent of human social norms. Here, we show that their results are better explained by data leakage: the models simply reproduce conventions they already encountered during pre-training. Despite the authors' mitigation measures, we provide multiple analyses demonstrating that the LLMs recognize the structure of the coordination game and recall its outcomes, rather than exhibit "emergent" conventions. Consequently, the observed behaviors are indistinguishable from memorization of the training corpus. We conclude by pointing to potential alternative strategies and reflecting more generally on the place of LLMs for social science models.