The mutual information of a stochastic binary channel: validity of the Replica Symmetry Ansatz

TL;DR

This paper investigates the mutual information in a neural network model, demonstrating a phase transition at a critical ratio of input to output units and evaluating the validity of the replica symmetric approximation.

Contribution

It provides an exact solution for the mutual information near zero ratio and analyzes the phase transition, challenging the assumptions of the replica symmetric ansatz.

Findings

Identifies a phase transition at a finite input-output ratio

Shows the replica symmetric solution is infinitely differentiable

Numerically validates the exact solution near zero ratio

Abstract

We calculate the mutual information (MI) of a two-layered neural network with noiseless, continuous inputs and binary, stochastic outputs under several assumptions on the synaptic efficiencies. The interesting regime corresponds to the limit where the number of both input and output units is large but their ratio is kept fixed at a value $\alpha$. We first present a solution for the MI using the replica technique with a replica symmetric (RS) ansatz. Then we find an exact solution for this quantity valid in a neighborhood of $\alpha = 0$. An analysis of this solution shows that the system must have a phase transition at some finite value of $\alpha$. This transition shows a singularity in the third derivative of the MI. As the RS solution turns out to be infinitely differentiable, it could be regarded as a smooth approximation to the MI. This is checked numerically in the validity domain of the exact solution.