Explicit mutual information for simple networks and neurons with lognormal activities

TL;DR

This paper derives exact formulas for mutual information in networks with heavy-tailed lognormal activities, revealing that such distributions can enhance information processing in neural systems.

Contribution

It provides the first analytical expressions for mutual information in networks with correlated lognormal activities, highlighting the impact of heavy tails on information capacity.

Findings

Mutual information can diverge with heavy-tailed distributions.

Neural activities and synaptic weights with lognormal distributions increase mutual information.

Heavy tails in neural variables may be evolutionarily advantageous for information processing.

Abstract

Networks with stochastic variables described by heavy tailed lognormal distribution are ubiquitous in nature, and hence they deserve an exact information-theoretic characterization. We derive analytical formulas for mutual information between elements of different networks with correlated lognormally distributed activities. In a special case, we find an explicit expression for mutual information between neurons when neural activities and synaptic weights are lognormally distributed, as suggested by experimental data. Comparison of this expression with the case when these two variables have short tails, reveals that mutual information with heavy tails for neurons and synapses is generally larger and can diverge for some finite variances in presynaptic firing rates and synaptic weights. This result suggests that evolution might prefer brains with heterogeneous dynamics to optimize information processing.