Statistical Consistency of Finite-dimensional Unregularized Linear Classification

TL;DR

This paper analyzes the statistical consistency of finite-dimensional unregularized linear classifiers, including logistic regression and boosting, showing they can achieve optimal risk with increasing sample size.

Contribution

It provides a finite-dimensional analysis of unregularized linear classifiers' consistency, encompassing various loss functions and boosting, with results on optimal risk convergence.

Findings

Linear classifiers are statistically consistent in finite dimensions.

Scaling complexity with sample size leads to near-optimal risk.

The analysis applies to logistic regression and boosting with various losses.

Abstract

This manuscript studies statistical properties of linear classifiers obtained through minimization of an unregularized convex risk over a finite sample. Although the results are explicitly finite-dimensional, inputs may be passed through feature maps; in this way, in addition to treating the consistency of logistic regression, this analysis also handles boosting over a finite weak learning class with, for instance, the exponential, logistic, and hinge losses. In this finite-dimensional setting, it is still possible to fit arbitrary decision boundaries: scaling the complexity of the weak learning class with the sample size leads to the optimal classification risk almost surely.