Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement

TL;DR

This paper demonstrates that transformers can explicitly implement classical reinforcement learning algorithms and guarantees their convergence under certain conditions, bridging mechanistic understanding and training dynamics.

Contribution

It provides the first provable implementation of policy improvement methods in transformers and analyzes their training dynamics with convergence guarantees.

Findings

Transformers can implement policy improvement algorithms like SARSA and actor-critic.

Gradient flow during training converges locally and exponentially to an optimal parameter manifold.

Trained transformers recover explicit parameter structures and perform well on unseen MDPs.

Abstract

We investigate the ability of transformers to perform in-context reinforcement learning (ICRL), where a model must infer and execute learning algorithms from trajectory data without parameter updates. We show that a linear self-attention transformer block can provably implement policy-improvement methods, including semi-gradient SARSA and actor-critic, via explicit parameter constructions. Beyond existence, we design a teacher-mimicking training procedure, analyze its gradient-flow dynamics, and establish the first convergence guarantee in the ICRL literature: under suitable richness conditions on the training MDP distribution, gradient flow converges locally and exponentially to an optimal parameter manifold corresponding to the desired RL update. Empirically, training transformers on randomly generated tabular MDPs confirms these predictions: the learned models recover the parameter structure of our explicit constructions and, when deployed on unseen MDPs, deliver strong in-context control performance. Together, these results illuminate how transformer architectures internalize and execute classical reinforcement learning algorithms in context, bridging mechanistic understanding and training dynamics in ICRL.