Entropy landscape of solutions in the binary perceptron problem

TL;DR

This paper investigates the structure of the solution space in the binary perceptron problem, revealing how it shrinks with more constraints and exhibits clustering and freezing phenomena near capacity.

Contribution

It provides a detailed entropy landscape analysis of the binary perceptron solutions using replica and message passing methods, connecting geometry to learning difficulty.

Findings

Solution space shrinks as constraints increase

Clustering and freezing coexist in the solution landscape

Numerical simulations confirm theoretical predictions

Abstract

The statistical picture of the solution space for a binary perceptron is studied. The binary perceptron learns a random classification of input random patterns by a set of binary synaptic weights. The learning of this network is difficult especially when the pattern (constraint) density is close to the capacity, which is supposed to be intimately related to the structure of the solution space. The geometrical organization is elucidated by the entropy landscape from a reference configuration and of solution-pairs separated by a given Hamming distance in the solution space. We evaluate the entropy at the annealed level as well as replica symmetric level and the mean field result is confirmed by the numerical simulations on single instances using the proposed message passing algorithms. From the first landscape (a random configuration as a reference), we see clearly how the solution space shrinks as more constraints are added. From the second landscape of solution-pairs, we deduce the coexistence of clustering and freezing in the solution space.