Weight space structure and analysis using a finite replica number in the Ising perceptron

TL;DR

This paper analyzes the structure of the weight space in the Ising perceptron using a finite replica number approach, revealing phase transitions in pattern separability and clustering behavior as the pattern-to-weight ratio varies.

Contribution

It introduces a novel application of the generating function of the partition function with finite replica number to study the weight space and pattern separability in the Ising perceptron.

Findings

Weight space is dominated by a large cluster and small clusters below a critical ratio.

Analyticity of the rate function changes at a specific ratio, indicating a phase transition.

Numerical experiments support the theoretical phase transition predictions.

Abstract

The weight space of the Ising perceptron in which a set of random patterns is stored is examined using the generating function of the partition function $\phi(n)=(1/N)\log [Z^n]$ as the dimension of the weight vector $N$ tends to infinity, where $Z$ is the partition function and $[ ... ]$ represents the configurational average. We utilize $\phi(n)$ for two purposes, depending on the value of the ratio $\alpha=M/N$, where $M$ is the number of random patterns. For $\alpha < \alpha_{\rm s}=0.833 ...$, we employ $\phi(n)$, in conjunction with Parisi's one-step replica symmetry breaking scheme in the limit of $n \to 0$, to evaluate the complexity that characterizes the number of disjoint clusters of weights that are compatible with a given set of random patterns, which indicates that, in typical cases, the weight space is equally dominated by a single large cluster of exponentially many weights and exponentially many small clusters of a single weight. For $\alpha > \alpha_{\rm s}$, on the other hand, $\phi(n)$ is used to assess the rate function of a small probability that a given set of random patterns is atypically separable by the Ising perceptrons. We show that the analyticity of the rate function changes at $\alpha = \alpha_{\rm GD}=1.245 ... $, which implies that the dominant configuration of the atypically separable patterns exhibits a phase transition at this critical ratio. Extensive numerical experiments are conducted to support the theoretical predictions.