Statistical mechanics of lossy compression using multilayer perceptrons

TL;DR

This paper applies statistical mechanics to analyze lossy compression with multilayer perceptrons, revealing how different network architectures approach theoretical limits of compression efficiency.

Contribution

It derives the rate distortion function for parity trees with multiple hidden units, expanding understanding of their compression capabilities.

Findings

Committee trees' lower bound on distortion decreases with more hidden units.

Parity trees with K >= 2 can theoretically reach the Shannon limit.

Theoretical analysis of code length and compression limits for different network architectures.

Abstract

Statistical mechanics is applied to lossy compression using multilayer perceptrons for unbiased Boolean messages. We utilize a tree-like committee machine (committee tree) and tree-like parity machine (parity tree) whose transfer functions are monotonic. For compression using committee tree, a lower bound of achievable distortion becomes small as the number of hidden units K increases. However, it cannot reach the Shannon bound even where K -> infty. For a compression using a parity tree with K >= 2 hidden units, the rate distortion function, which is known as the theoretical limit for compression, is derived where the code length becomes infinity.