The Topological Trouble With Transformers

TL;DR

This paper discusses the limitations of feedforward transformers in dynamic state tracking and advocates for recurrent architectures to improve temporal cognition in models.

Contribution

It introduces a taxonomy of recurrent transformer architectures and suggests research directions for better state integration in foundation models.

Findings

Feedforward transformers struggle with dynamic state tracking due to depth limitations.

Recurrent architectures can better support temporal cognition by maintaining implicit activation dynamics.

The paper outlines a taxonomy categorizing recurrent transformer designs by recurrence axis and input-to-recurrence ratio.

Abstract

Transformers encode structure in sequences via an expanding contextual history. However, their purely feedforward architecture fundamentally limits dynamic state tracking. State tracking -- the iterative updating of latent variables reflecting an evolving environment -- involves inherently sequential dependencies that feedforward networks struggle to maintain. Consequently, feedforward models push evolving state representations deeper into their layer stack with each new input step, rendering information inaccessible in shallow layers and ultimately exhausting the model's depth. While this depth limit can be bypassed by dynamic depth models and by explicit or latent thinking that externalizes state representations, these solutions are computationally and memory inefficient. In this article, we argue that temporally extended cognition requires refocusing from explicit thought traces to implicit activation dynamics via recurrent architectures. We introduce a taxonomy of recurrent and continuous-thought transformer architectures, categorizing them by their recurrence axis (depth versus step) and their ratio of input tokens to recurrence steps. Finally, we outline promising research directions, including enhanced state-space models and coarse-grained recurrence, to better integrate state tracking into modern foundation models.