Aligned and oblique dynamics in recurrent neural networks

TL;DR

This paper investigates how recurrent neural networks exhibit aligned or oblique dynamics relative to output variables, revealing how these regimes influence robustness and heterogeneity, with implications for understanding neural activity.

Contribution

The study introduces a geometrical framework for understanding RNN dynamics, identifying distinct aligned and oblique regimes and their control via readout weights, specific to recurrent networks.

Findings

Oblique regimes are more noise-robust and heterogeneous.

The choice of readout weights influences the dynamical regime.

Oblique dynamics are specific to recurrent networks, not feedforward.

Abstract

The relation between neural activity and behaviorally relevant variables is at the heart of neuroscience research. When strong, this relation is termed a neural representation. There is increasing evidence, however, for partial dissociations between activity in an area and relevant external variables. While many explanations have been proposed, a theoretical framework for the relationship between external and internal variables is lacking. Here, we utilize recurrent neural networks (RNNs) to explore the question of when and how neural dynamics and the network's output are related from a geometrical point of view. We find that training RNNs can lead to two dynamical regimes: dynamics can either be aligned with the directions that generate output variables, or oblique to them. We show that the choice of readout weight magnitude before training can serve as a control knob between the regimes, similar to recent findings in feedforward networks. These regimes are functionally distinct. Oblique networks are more heterogeneous and suppress noise in their output directions. They are furthermore more robust to perturbations along the output directions. Crucially, the oblique regime is specific to recurrent (but not feedforward) networks, arising from dynamical stability considerations. Finally, we show that tendencies towards the aligned or the oblique regime can be dissociated in neural recordings. Altogether, our results open a new perspective for interpreting neural activity by relating network dynamics and their output.