Task Ecologies and the Evolution of World-Tracking Representations in Large Language Models

TL;DR

This paper develops a theoretical framework to understand when language models develop world-tracking representations, analyzing their ecological veridicality and failure modes through a formal decomposition.

Contribution

It introduces a precise notion of ecological veridicality, characterizes when fixed encodings suffice, and validates the theory with experiments on small language models.

Findings

The excess cross-entropy vanishes if encoding preserves training ecology's equivalence classes.

Frozen transformers satisfy the fixed-encoding analysis, while in-context learning does not.

Models can still incur positive excess on refined deployment ecologies, indicating limitations.

Abstract

We study language models as evolving model organisms and ask when autoregressive next-token learning selects for world-tracking representations. For any encoding of latent world states, the Bayes-optimal next-token cross-entropy decomposes into the irreducible conditional entropy plus a Jensen--Shannon excess term. That excess vanishes if and only if the encoding preserves the training ecology's equivalence classes. This yields a precise notion of ecological veridicality for language models and identifies the minimum-complexity zero-excess solution as the quotient partition by training equivalence. We then determine when this fixed-encoding analysis applies to transformer families: frozen dense and frozen Mixture-of-Experts transformers satisfy it, in-context learning does not enlarge the model's separation set, and per-task adaptation breaks the premise. The framework predicts two characteristic failure modes: simplicity pressure preferentially removes low-gain distinctions, and training-optimal models can still incur positive excess on deployment ecologies that refine the training ecology. A conditional dynamic extension shows how inter-model selection and post-training can recover such gap distinctions under explicit heredity, variation, and selection assumptions. Exact finite-ecology checks and controlled microgpt experiments validate the static decomposition, split-merge threshold, off-ecology failure pattern, and two-ecology rescue mechanism in a regime where the relevant quantities are directly observable. The goal is not to model frontier systems at scale, but to use small language models as laboratory organisms for theory about representational selection.