Tight bounds for maximum $\ell_1$-margin classifiers

TL;DR

This paper establishes tight bounds on the prediction error of the maximum -margin classifier in high-dimensional settings, revealing its limitations and benign overfitting behavior.

Contribution

It provides the first rigorous analysis of the maximum -margin classifier's error bounds, showing they do not adapt to sparse ground truths and demonstrating benign overfitting in noisy scenarios.

Findings

Prediction error bounds match existing rates of /3 for sparse ground truths.

Error vanishes at a rate of 1/(/d/n) in noisy settings.

First demonstration of benign overfitting for maximum -margin classifiers.

Abstract

Popular iterative algorithms such as boosting methods and coordinate descent on linear models converge to the maximum $\ell_1$-margin classifier, a.k.a. sparse hard-margin SVM, in high dimensional regimes where the data is linearly separable. Previous works consistently show that many estimators relying on the $\ell_1$-norm achieve improved statistical rates for hard sparse ground truths. We show that surprisingly, this adaptivity does not apply to the maximum $\ell_1$-margin classifier for a standard discriminative setting. In particular, for the noiseless setting, we prove tight upper and lower bounds for the prediction error that match existing rates of order $\frac{\|w^*\|_1^{2/3}}{n^{1/3}}$ for general ground truths. To complete the picture, we show that when interpolating noisy observations, the error vanishes at a rate of order $\frac{1}{\sqrt{\log(d/n)}}$. We are therefore first to show benign overfitting for the maximum $\ell_1$-margin classifier.