Probing the relationship between linear dynamical systems and low-rank recurrent neural network models

TL;DR

This paper explores the theoretical relationship between linear dynamical systems and low-rank recurrent neural networks, clarifying when one model can be transformed into the other and highlighting their fundamental differences.

Contribution

It provides a formal analysis of the conditions under which latent LDS models and low-rank RNNs can be converted into each other, revealing their limitations and connections.

Findings

Latent LDS models can only be converted to RNNs in specific limit cases.

Linear RNNs can be mapped onto LDS models with latent dimensionality at most twice the RNN's rank.

The non-Markovian property of LDS models limits their conversion to RNNs.

Abstract

A large body of work has suggested that neural populations exhibit low-dimensional dynamics during behavior. However, there are a variety of different approaches for modeling low-dimensional neural population activity. One approach involves latent linear dynamical system (LDS) models, in which population activity is described by a projection of low-dimensional latent variables with linear dynamics. A second approach involves low-rank recurrent neural networks (RNNs), in which population activity arises directly from a low-dimensional projection of past activity. Although these two modeling approaches have strong similarities, they arise in different contexts and tend to have different domains of application. Here we examine the precise relationship between latent LDS models and linear low-rank RNNs. When can one model class be converted to the other, and vice versa? We show that latent LDS models can only be converted to RNNs in specific limit cases, due to the non-Markovian property of latent LDS models. Conversely, we show that linear RNNs can be mapped onto LDS models, with latent dimensionality at most twice the rank of the RNN.