Origin of the computational hardness for learning with binary synapses

TL;DR

This paper analytically investigates the structure of the weight space in binary perceptrons, revealing isolated solutions that explain the computational hardness and glassy dynamics observed in learning algorithms.

Contribution

It derives the Franz-Parisi potential for the binary perceptron, unveiling the geometrical organization of weight space as isolated solutions rather than clusters.

Findings

Weight space consists of isolated solutions.

Solutions are far apart, leading to glassy dynamics.

Explains the difficulty of finding solutions in practice.

Abstract

Supervised learning in a binary perceptron is able to classify an extensive number of random patterns by a proper assignment of binary synaptic weights. However, to find such assignments in practice, is quite a nontrivial task. The relation between the weight space structure and the algorithmic hardness has not yet been fully understood. To this end, we analytically derive the Franz-Parisi potential for the binary preceptron problem, by starting from an equilibrium solution of weights and exploring the weight space structure around it. Our result reveals the geometrical organization of the weight space\textemdash the weight space is composed of isolated solutions, rather than clusters of exponentially many close-by solutions. The point-like clusters far apart from each other in the weight space explain the previously observed glassy behavior of stochastic local search heuristics.