Replica Symmetry Breaking and the Kuhn-Tucker Cavity Method in simple and multilayer Perceptrons

TL;DR

This paper applies the Kuhn-Tucker cavity method to analyze stability learning in simple and multilayer perceptrons, revealing the importance of replica symmetry breaking for accurate capacity estimation.

Contribution

It introduces a cavity method approach to detect replica symmetry breaking in perceptrons and multilayer networks, improving understanding of their storage limits.

Findings

Cavity solutions deviate from replica symmetric solutions indicating symmetry breaking.

The cavity method underestimates the networks' storage capacity.

Replica symmetry breaking is necessary for accurate capacity estimation.

Abstract

Within a Kuhn-Tucker cavity method introduced in a former paper, we study optimal stability learning for situations, where in the replica formalism the replica symmetry may be broken, namely (i) the case of a simple perceptron above the critical loading, and (ii) the case of two-layer AND-perceptrons, if one learns with maximal stability. We find that the deviation of our cavity solution from the replica symmetric one in these cases is a clear indication of the necessity of replica symmetry breaking. In any case the cavity solution tends to underestimate the storage capabilities of the networks.