Sharp concentration of uniform generalization errors in binary linear classification

TL;DR

This paper investigates how uniformly the generalization errors in binary linear classification concentrate around their expected values, providing sharp bounds and broad asymptotic convergence results.

Contribution

It introduces new concentration bounds for generalization errors using isoperimetric inequalities and establishes uniform laws of large numbers in high-dimensional settings.

Findings

Concentration bounds are sharp up to constants for well-balanced labels.

Almost sure convergence of errors occurs in high-dimensional regimes.

Uniform laws of large numbers hold under dimension-free conditions.

Abstract

We examine the concentration of uniform generalization errors around their expectation in binary linear classification problems via an isoperimetric argument. In particular, we establish Poincar\'{e} and log-Sobolev inequalities for the joint distribution of the output labels and the label-weighted input vectors, which we apply to derive concentration bounds. The derived concentration bounds are sharp up to moderate multiplicative constants by those under well-balanced labels. In asymptotic analysis, we also show that almost sure convergence of uniform generalization errors to their expectation occurs in very broad settings, such as proportionally high-dimensional regimes. Using this convergence, we establish uniform laws of large numbers under dimension-free conditions.