Modeling Language as a Sequence of Thoughts

TL;DR

The paper introduces the Thought Gestalt (TG) model, a recurrent transformer that models language at token and sentence levels, improving relational generalization and efficiency by mimicking human event-like representations.

Contribution

The TG model is a novel recurrent transformer architecture that incorporates sentence-level thought states with shared parameters, enhancing language modeling and generalization.

Findings

TG improves data and parameter efficiency over GPT-2.

TG reduces relational-direction errors in generalization tasks.

Scaling experiments show TG requires fewer resources to achieve similar performance.

Abstract

Transformer language models can generate strikingly natural text by modeling language as a sequence of tokens, but by relying primarily on surface-level co-occurrence statistics they fail to form globally consistent latent representations of entities and events, which contributes to poor relational generalization (the reversal curse), contextualization errors, and data inefficiency. Cognitive science, by contrast, shows that human comprehension converts linguistic input into compact, event-like representations that persist in memory while verbatim form is short-lived. Motivated by these findings, we introduce the Thought Gestalt (TG) model, a recurrent transformer that models language at two levels of abstraction: tokens and sentence-level "thought" states. TG generates one sentence at a time while cross-attending to a working memory of prior sentence representations. Token and sentence representations are generated using a shared stack of transformer blocks and trained with a single objective, next-token prediction loss. By retaining the computation graph of sentence representations written to working memory, gradients from future token losses flow backward through cross-attention to optimize the parameters that generate earlier sentence vectors. In scaling experiments, TG consistently improves data and parameter efficiency compared to matched GPT-2 runs and other baselines, with scaling fits indicating GPT-2 requires ~5-8% more data and ~33-42% more parameters to match TG's test loss. TG also reduces errors in relational-direction generalization on a father-son reversal curse probe.