Learning State-Tracking from Code Using Linear RNNs

TL;DR

This paper demonstrates that linear RNNs excel at state-tracking in code, especially in permutation tasks, while Transformers struggle, highlighting the importance of model architecture in sequence understanding.

Contribution

It introduces a method to convert permutation composition into code with REPL traces and shows linear RNNs outperform Transformers in this setting.

Findings

Linear RNNs excel at state-tracking in code tasks.

Transformers fail to perform well in permutation composition tasks.

Tracking states in code is difficult due to partial observability.

Abstract

Over the last years, state-tracking tasks, particularly permutation composition, have become a testbed to understand the limits of sequence models architectures like Transformers and RNNs (linear and non-linear). However, these are often sequence-to-sequence tasks: learning to map actions (permutations) to states, which is incompatible with the next-token prediction setting commonly used to train language models. We address this gap by converting permutation composition into code via REPL traces that interleave state-reveals through prints and variable transformations. We show that linear RNNs capable of state-tracking excel also in this setting, while Transformers still fail. Motivated by this representation, we investigate why tracking states in code is generally difficult: actions are not always fully observable. We frame this as tracking the state of a probabilistic finite-state automaton with deterministic state reveals and show that linear RNNs can be worse than non-linear RNNs at tracking states in this setup.