Statistical mechanics of lossy compression for non-monotonic multilayer perceptrons

TL;DR

This paper analyzes a lossy compression scheme for biased Boolean messages using statistical mechanics, demonstrating that certain non-monotonic multilayer perceptrons can achieve Shannon limit performance.

Contribution

It introduces a lossy compression scheme employing non-monotonic tree-like perceptrons and analyzes its optimality and stability using replica methods.

Findings

Scheme saturates Shannon bound

Non-monotonic transfer functions improve performance

Replica symmetric solution is AT stable

Abstract

A lossy data compression scheme for uniformly biased Boolean messages is investigated via statistical mechanics techniques. We utilize tree-like committee machine (committee tree) and tree-like parity machine (parity tree) whose transfer functions are non-monotonic. The scheme performance at the infinite code length limit is analyzed using the replica method. Both committee and parity treelike networks are shown to saturate the Shannon bound. The AT stability of the Replica Symmetric solution is analyzed, and the tuning of the non-monotonic transfer function is also discussed.