Adaptive Loops and Memory in Transformers: Think Harder or Know More?

TL;DR

This paper introduces transformer models with adaptive looping and gated memory banks, enhancing reasoning and knowledge retention, especially in mathematical and commonsense tasks, outperforming traditional models with similar computational costs.

Contribution

It proposes a novel combination of adaptive per-layer looping and memory banks in transformers, improving reasoning and task performance over existing models.

Findings

Looping benefits mathematical reasoning.

Memory banks improve commonsense task performance.

Combined mechanisms outperform iso-FLOP baselines.

Abstract

Chain-of-thought (CoT) prompting enables reasoning in language models but requires explicit verbalization of intermediate steps. Looped transformers offer an alternative by iteratively refining representations within hidden states. This parameter efficiency comes at a cost, as looped models lack the storage capacity of deeper models which use unique weights per layer. In this work, we investigate transformer models that feature both adaptive per-layer looping, where each transformer block learns to iterate its hidden state via a learned halting mechanism, and gated memory banks, that provide additional learned storage. We find that looping primarily benefits mathematical reasoning, while memory banks help recover performance on commonsense tasks compared to parameter and FLOP matched models. Combining both mechanisms yields a model that outperforms an iso-FLOP baseline, with three times the number of layers, across math benchmarks. Analysis of model internals reveals layer specialization: early layers learn to loop minimally and access memory sparingly, while later layers do both more heavily.