Stochastic Pooling Networks

TL;DR

This paper introduces stochastic pooling networks (SPNs), a model for sensor systems where redundancy, noise, and compression interact, analyzed through information theory to show how pooling reduces redundancy without losing mutual information.

Contribution

The paper defines SPNs and demonstrates their properties, showing how pooling can reduce redundancy while preserving information across various physical and biological systems.

Findings

Pooling in SPNs reduces redundancy without significant information loss

SPNs can be modeled with nodes as noisy quantizers or Poisson processes

Emergent properties of SPNs are discussed with case studies

Abstract

We introduce and define the concept of a stochastic pooling network (SPN), as a model for sensor systems where redundancy and two forms of 'noise' -- lossy compression and randomness -- interact in surprising ways. Our approach to analyzing SPNs is information theoretic. We define an SPN as a network with multiple nodes that each produce noisy and compressed measurements of the same information. An SPN must combine all these measurements into a single further compressed network output, in a way dictated solely by naturally occurring physical properties -- i.e. pooling -- and yet causes no (or negligible) reduction in mutual information. This means SPNs exhibit redundancy reduction as an emergent property of pooling. The SPN concept is applicable to examples in biological neural coding, nano-electronics, distributed sensor networks, digital beamforming arrays, image processing, multiaccess communication networks and social networks. In most cases the randomness is assumed to be unavoidably present rather than deliberately introduced. We illustrate the central properties of SPNs for several case studies, where pooling occurs by summation, including nodes that are noisy scalar quantizers, and nodes with conditionally Poisson statistics. Other emergent properties of SPNs and some unsolved problems are also briefly discussed.