Short-Term Memory in Orthogonal Neural Networks

TL;DR

This paper investigates the memory capacity of linear recurrent neural networks with orthogonal connectivity, demonstrating how their ability to store long sequences scales with system size.

Contribution

It provides a theoretical analysis of memory capacity in orthogonal neural networks, including explicit calculations for different connectivity types.

Findings

Memory capacity scales with system size.

Orthogonal networks can store long sequences effectively.

Capacity depends on the type of connectivity matrix.

Abstract

We study the ability of linear recurrent networks obeying discrete time dynamics to store long temporal sequences that are retrievable from the instantaneous state of the network. We calculate this temporal memory capacity for both distributed shift register and random orthogonal connectivity matrices. We show that the memory capacity of these networks scales with system size.