Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks

TL;DR

This paper investigates how recurrent neural networks learn to integrate multiple temporal signals, revealing that their internal states reside on low-dimensional manifolds that encode information about these signals, with implications for understanding neural computation.

Contribution

It provides a combined analytical and numerical analysis of RNNs' learning dynamics, showing how low-dimensional manifolds support multiplexed integration of signals, and relates these findings to neuroscience concepts.

Findings

Internal states of RNNs are close to a low-dimensional manifold.

The shape of the manifold depends on the activation function.

Neurons encode information about multiple integrals simultaneously.

Abstract

We study the learning dynamics and the representations emerging in Recurrent Neural Networks trained to integrate one or multiple temporal signals. Combining analytical and numerical investigations, we characterize the conditions under which a RNN with n neurons learns to integrate D(n) scalar signals of arbitrary duration. We show, both for linear and ReLU neurons, that its internal state lives close to a D-dimensional manifold, whose shape is related to the activation function. Each neuron therefore carries, to various degrees, information about the value of all integrals. We discuss the deep analogy between our results and the concept of mixed selectivity forged by computational neuroscientists to interpret cortical recordings.