Regularized Linear Regression for Binary Classification

TL;DR

This paper systematically studies how different regularization techniques affect linear classifiers in noisy binary classification, revealing that ridge improves accuracy, $ ext{l}_1$ induces sparsity, and $ ext{l}_ extinfty$ can compress weights with minimal performance loss.

Contribution

It provides a rigorous analysis of regularization effects in over-parameterized linear classifiers trained on noisy data, highlighting practical benefits of sparsity and weight compression.

Findings

Ridge regression consistently reduces classification error.

$ ext{l}_1$ regularization induces sparsity without performance loss.

$ ext{l}_ extinfty$ regularization leads to weight concentration and effective compression.

Abstract

Regularized linear regression is a promising approach for binary classification problems in which the training set has noisy labels since the regularization term can help to avoid interpolating the mislabeled data points. In this paper we provide a systematic study of the effects of the regularization strength on the performance of linear classifiers that are trained to solve binary classification problems by minimizing a regularized least-squares objective. We consider the over-parametrized regime and assume that the classes are generated from a Gaussian Mixture Model (GMM) where a fraction $c<\frac{1}{2}$ of the training data is mislabeled. Under these assumptions, we rigorously analyze the classification errors resulting from the application of ridge, $\ell_1$, and $\ell_\infty$ regression. In particular, we demonstrate that ridge regression invariably improves the classification error. We prove that $\ell_1$ regularization induces sparsity and observe that in many cases one can sparsify the solution by up to two orders of magnitude without any considerable loss of performance, even though the GMM has no underlying sparsity structure. For $\ell_\infty$ regularization we show that, for large enough regularization strength, the optimal weights concentrate around two values of opposite sign. We observe that in many cases the corresponding "compression" of each weight to a single bit leads to very little loss in performance. These latter observations can have significant practical ramifications.