Learning Dynamics of RNNs in Closed-Loop Environments

TL;DR

This paper develops a mathematical theory for the learning dynamics of linear RNNs in closed-loop environments, revealing distinct trajectories from open-loop training and emphasizing the importance of closed-loop modeling for biological plausibility.

Contribution

It introduces a novel analytical framework for understanding how linear RNNs learn in closed-loop settings, contrasting with open-loop paradigms.

Findings

Closed-loop RNNs follow different learning trajectories than open-loop RNNs.

The dynamics involve a balance between short-term policy improvement and long-term stability.

Framework successfully applied to a realistic motor control task.

Abstract

Recurrent neural networks (RNNs) trained on neuroscience-inspired tasks offer powerful models of brain computation. However, typical training paradigms rely on open-loop, supervised settings, whereas real-world learning unfolds in closed-loop environments. Here, we develop a mathematical theory describing the learning dynamics of linear RNNs trained in closed-loop contexts. We first demonstrate that two otherwise identical RNNs, trained in either closed- or open-loop modes, follow markedly different learning trajectories. To probe this divergence, we analytically characterize the closed-loop case, revealing distinct stages aligned with the evolution of the training loss. Specifically, we show that the learning dynamics of closed-loop RNNs, in contrast to open-loop ones, are governed by an interplay between two competing objectives: short-term policy improvement and long-term stability of the agent-environment interaction. Finally, we apply our framework to a realistic motor control task, highlighting its broader applicability. Taken together, our results underscore the importance of modeling closed-loop dynamics in a biologically plausible setting.