Context-dependent representation in recurrent neural networks

TL;DR

This paper investigates how recurrent neural networks encode past information, revealing an optimal level of synaptic heterogeneity and emphasizing the role of network topology in context-dependent representations.

Contribution

It introduces a measure for context dependence in neural networks and analyzes how synaptic heterogeneity and network symmetry influence memory performance.

Findings

Optimal synaptic heterogeneity enhances memory capacity.

Network symmetry affects the dependence on past context.

Theoretical and numerical analysis confirms the interplay between non-linearities and connectivity.

Abstract

In order to assess the short-term memory performance of non-linear random neural networks, we introduce a measure to quantify the dependence of a neural representation upon the past context. We study this measure both numerically and theoretically using the mean-field theory for random neural networks, showing the existence of an optimal level of synaptic weights heterogeneity. We further investigate the influence of the network topology, in particular the symmetry of reciprocal synaptic connections, on this measure of context dependence, revealing the importance of considering the interplay between non-linearities and connectivity structure.