Novel and Efficient Approximations for Zero-One Loss of Linear Classifiers

TL;DR

This paper introduces novel smooth approximation functions for the expected error and ranking loss of linear classifiers, enabling faster optimization with comparable or improved accuracy.

Contribution

It proposes data distribution moment-based approximations for linear classifier loss functions, reducing computational complexity and improving training efficiency.

Findings

Approximation functions depend on data moments, not data size.

The approach achieves similar or better accuracy and AUC.

Optimization is significantly faster than state-of-the-art methods.

Abstract

The predictive quality of machine learning models is typically measured in terms of their (approximate) expected prediction accuracy or the so-called Area Under the Curve (AUC). Minimizing the reciprocals of these measures are the goals of supervised learning. However, when the models are constructed by the means of empirical risk minimization (ERM), surrogate functions such as the logistic loss or hinge loss are optimized instead. In this work, we show that in the case of linear predictors, the expected error and the expected ranking loss can be effectively approximated by smooth functions whose closed form expressions and those of their first (and second) order derivatives depend on the first and second moments of the data distribution, which can be precomputed. Hence, the complexity of an optimization algorithm applied to these functions does not depend on the size of the training data. These approximation functions are derived under the assumption that the output of the linear classifier for a given data set has an approximately normal distribution. We argue that this assumption is significantly weaker than the Gaussian assumption on the data itself and we support this claim by demonstrating that our new approximation is quite accurate on data sets that are not necessarily Gaussian. We present computational results that show that our proposed approximations and related optimization algorithms can produce linear classifiers with similar or better test accuracy or AUC, than those obtained using state-of-the-art approaches, in a fraction of the time.