Binary perceptron: efficient algorithms can find solutions in a rare well-connected cluster

TL;DR

This paper proves that at low constraint densities, solutions to binary perceptrons form large, well-connected clusters that can be efficiently found by algorithms, contrasting with the isolated nature of typical solutions.

Contribution

It formally establishes the existence of large, connected solution clusters in binary perceptrons and demonstrates that simple algorithms can efficiently find solutions within these clusters.

Findings

Existence of a subdominant connected cluster of solutions at low density.

Efficient multiscale majority algorithm can find solutions in these clusters.

Clusters of linear diameter exist near the critical threshold.

Abstract

It was recently shown that almost all solutions in the symmetric binary perceptron are isolated, even at low constraint densities, suggesting that finding typical solutions is hard. In contrast, some algorithms have been shown empirically to succeed in finding solutions at low density. This phenomenon has been justified numerically by the existence of subdominant and dense connected regions of solutions, which are accessible by simple learning algorithms. In this paper, we establish formally such a phenomenon for both the symmetric and asymmetric binary perceptrons. We show that at low constraint density (equivalently for overparametrized perceptrons), there exists indeed a subdominant connected cluster of solutions with almost maximal diameter, and that an efficient multiscale majority algorithm can find solutions in such a cluster with high probability, settling in particular an open problem posed by Perkins-Xu '21. In addition, even close to the critical threshold, we show that there exist clusters of linear diameter for the symmetric perceptron, as well as for the asymmetric perceptron under additional assumptions.