Stability of the replica symmetric solution for the information conveyed by by a neural network

TL;DR

This paper analyzes the stability of the replica-symmetric solution in neural networks, showing it remains stable under most conditions except for low noise levels, with stability influenced by threshold and sparseness.

Contribution

It provides a stability analysis of the replica-symmetric solution in neural networks considering noise, threshold, and sparseness effects, which was not previously detailed.

Findings

Replica-symmetric solution is stable for most noise levels.

Instability region depends on threshold and sparseness.

Distributed patterns have larger unstable regions than sparse patterns.

Abstract

The information that a pattern of firing in the output layer of a feedforward network of threshold-linear neurons conveys about the network's inputs is considered. A replica-symmetric solution is found to be stable for all but small amounts of noise. The region of instability depends on the contribution of the threshold and the sparseness: for distributed pattern distributions, the unstable region extends to higher noise variances than for very sparse distributions, for which it is almost nonexistant.