Capacity-resolution trade-off in the optimal learning of multiple low-dimensional manifolds by attractor neural networks

TL;DR

This paper investigates how recurrent neural networks can learn multiple low-dimensional manifolds, revealing a capacity-resolution trade-off that depends on the error tolerance and manifold dimension, using advanced theoretical tools.

Contribution

It extends Gardner's classical theory to correlated patterns, quantifies the capacity-resolution trade-off, and demonstrates RNNs' ability to store many manifolds with high spatial resolution.

Findings

Capacity decreases as |log(epsilon)|^(-D) with error epsilon and manifold dimension D

RNNs can embed a large number of manifolds relative to neurons

Analytical results derived using statistical mechanics and random matrix theory

Abstract

Recurrent neural networks (RNN) are powerful tools to explain how attractors may emerge from noisy, high-dimensional dynamics. We study here how to learn the ~N^(2) pairwise interactions in a RNN with N neurons to embed L manifolds of dimension D << N. We show that the capacity, i.e. the maximal ratio L/N, decreases as |log(epsilon)|^(-D), where epsilon is the error on the position encoded by the neural activity along each manifold. Hence, RNN are flexible memory devices capable of storing a large number of manifolds at high spatial resolution. Our results rely on a combination of analytical tools from statistical mechanics and random matrix theory, extending Gardner's classical theory of learning to the case of patterns with strong spatial correlations.