Low Tensor Rank Learning of Neural Dynamics

TL;DR

This paper demonstrates that neural network weights and their evolution during learning are low-tensor-rank, revealing a low-dimensional structure in neural dynamics that can be mathematically characterized and observed in biological data.

Contribution

It introduces the concept of low-tensor-rank learning in neural networks and provides mathematical bounds, supported by empirical evidence from neural recordings and RNN training.

Findings

Weights are low-tensor-rank throughout learning

Low-tensor-rank weights emerge naturally in low-dimensional tasks

Supports reverse engineering of neural dynamics from data

Abstract

Learning relies on coordinated synaptic changes in recurrently connected populations of neurons. Therefore, understanding the collective evolution of synaptic connectivity over learning is a key challenge in neuroscience and machine learning. In particular, recent work has shown that the weight matrices of task-trained RNNs are typically low rank, but how this low rank structure unfolds over learning is unknown. To address this, we investigate the rank of the 3-tensor formed by the weight matrices throughout learning. By fitting RNNs of varying rank to large-scale neural recordings during a motor learning task, we find that the inferred weights are low-tensor-rank and therefore evolve over a fixed low-dimensional subspace throughout the entire course of learning. We next validate the observation of low-tensor-rank learning on an RNN trained to solve the same task. Finally, we present a set of mathematical results bounding the matrix and tensor ranks of gradient descent learning dynamics which show that low-tensor-rank weights emerge naturally in RNNs trained to solve low-dimensional tasks. Taken together, our findings provide insight on the evolution of population connectivity over learning in both biological and artificial neural networks, and enable reverse engineering of learning-induced changes in recurrent dynamics from large-scale neural recordings.