An Analysis of Loss Functions for Binary Classification and Regression

TL;DR

This paper investigates the theoretical properties of margin-based loss functions in binary classification and regression, providing new characterizations, a novel loss function, and insights into existing algorithms like AdaBoost.

Contribution

It introduces a simple characterization for conformable loss functions, constructs a new Huber-type loss, and links margin-based losses to logistic residuals for better understanding.

Findings

A new Huber-type loss function for logistic models is proposed.

Margin-based loss functions are shown to relate to squared standardized logistic residuals.

Insights into the sensitivity of exponential loss to outliers and the AdaBoost algorithm are provided.

Abstract

This paper explores connections between margin-based loss functions and consistency in binary classification and regression applications. It is shown that a large class of margin-based loss functions for binary classification/regression result in estimating scores equivalent to log-likelihood scores weighted by an even function. A simple characterization for conformable (consistent) loss functions is given, which allows for straightforward comparison of different losses, including exponential loss, logistic loss, and others. The characterization is used to construct a new Huber-type loss function for the logistic model. A simple relation between the margin and standardized logistic regression residuals is derived, demonstrating that all margin-based loss can be viewed as loss functions of squared standardized logistic regression residuals. The relation provides new, straightforward interpretations for exponential and logistic loss, and aids in understanding why exponential loss is sensitive to outliers. In particular, it is shown that minimizing empirical exponential loss is equivalent to minimizing the sum of squared standardized logistic regression residuals. The relation also provides new insight into the AdaBoost algorithm.