Simplified derivations for high-dimensional convex learning problems

TL;DR

This paper introduces simplified, unified derivations for high-dimensional convex learning problems in machine learning and neuroscience, clarifying their structure and capacity limits using a cavity approach.

Contribution

It provides concise, non-replica derivations for perceptron and kernel ridge regression problems, revealing underlying similarities and symmetries.

Findings

Unified analysis of perceptron and kernel methods

Identification of symmetry in perceptron capacity

Simplified derivations for complex high-dimensional problems

Abstract

Statistical-physics calculations in machine learning and theoretical neuroscience often involve lengthy derivations that obscure physical interpretation. Here, we give concise, non-replica derivations of several key results and highlight their underlying similarities. In particular, using a cavity approach, we analyze three high-dimensional learning problems: perceptron classification of points, perceptron classification of manifolds, and kernel ridge regression. These problems share a common structure--a bipartite system of interacting feature and datum variables--enabling a unified analysis. Furthermore, for perceptron-capacity problems, we identify a symmetry that allows derivation of correct capacities through a naive method.