Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regime

TL;DR

This paper provides finite-sample bounds on the population risk of maximum margin classifiers in overparameterized linear models, demonstrating near-optimal performance even with noisy data.

Contribution

It offers the first finite-sample analysis of maximum margin classifiers in the overparameterized regime with noisy data, including adversarial label corruption.

Findings

Maximum margin classifiers achieve near-optimal risk with sufficient overparameterization.

The analysis applies to data with normal class-conditional distributions and adversarial label noise.

Bounds hold even when training data includes misclassification noise.

Abstract

We prove bounds on the population risk of the maximum margin algorithm for two-class linear classification. For linearly separable training data, the maximum margin algorithm has been shown in previous work to be equivalent to a limit of training with logistic loss using gradient descent, as the training error is driven to zero. We analyze this algorithm applied to random data including misclassification noise. Our assumptions on the clean data include the case in which the class-conditional distributions are standard normal distributions. The misclassification noise may be chosen by an adversary, subject to a limit on the fraction of corrupted labels. Our bounds show that, with sufficient over-parameterization, the maximum margin algorithm trained on noisy data can achieve nearly optimal population risk.