Finite size scaling of the bayesian perceptron

TL;DR

This paper investigates the finite size effects of the Bayesian perceptron, combining numerical simulations with theoretical analysis to understand its generalization properties and stability near error-free solutions.

Contribution

It provides a detailed numerical and theoretical analysis of the Bayesian perceptron's finite size scaling and generalization error behavior.

Findings

Optimal generalizer is near the boundary of error-free solutions.

Finite size corrections are negative with two scaling regimes.

Variance of generalization error vanishes as input size grows.

Abstract

We study numerically the properties of the bayesian perceptron through a gradient descent on the optimal cost function. The theoretical distribution of stabilities is deduced. It predicts that the optimal generalizer lies close to the boundary of the space of (error-free) solutions. The numerical simulations are in good agreement with the theoretical distribution. The extrapolation of the generalization error to infinite input space size agrees with the theoretical results. Finite size corrections are negative and exhibit two different scaling regimes, depending on the training set size. The variance of the generalization error vanishes for $N \rightarrow \infty$ confirming the property of self-averaging.