Storage capacity of perceptron with variable selection

TL;DR

This paper explores the storage capacity of perceptrons with variable selection, demonstrating how optimal variable choice can surpass traditional bounds and help distinguish genuine data structure from chance correlations.

Contribution

It introduces a replica-based method to enumerate variable combinations that enable perfect classification, surpassing the Cover-Gardner bound.

Findings

Optimal variable selection improves storage capacity beyond traditional bounds.

The method distinguishes true data structure from spurious correlations.

Provides a quantitative criterion for data structure validation.

Abstract

A central challenge in machine learning is to distinguish genuine structure from chance correlations in high-dimensional data. In this work, we address this issue for the perceptron, a foundational model of neural computation. Specifically, we investigate the relationship between the pattern load $\alpha$ and the variable selection ratio $\rho$ for which a simple perceptron can perfectly classify $P = \alpha N$ random patterns by optimally selecting $M = \rho N$ variables out of $N$ variables. While the Cover--Gardner theory establishes that a random subset of $\rho N$ dimensions can separate $\alpha N$ random patterns if and only if $\alpha < 2\rho$, we demonstrate that optimal variable selection can surpass this bound by developing a method, based on the replica method from statistical mechanics, for enumerating the combinations of variables that enable perfect pattern classification. This not only provides a quantitative criterion for distinguishing true structure in the data from spurious regularities, but also yields the storage capacity of associative memory models with sparse asymmetric couplings.