An Alternative Probabilistic Interpretation of the Huber Loss

TL;DR

This paper introduces an alternative probabilistic interpretation of the Huber loss, linking it to KL divergence between Laplace distributions, which aids in selecting hyper-parameters based on noise estimation in data.

Contribution

It proposes a new probabilistic perspective that relates the Huber loss transition point to noise distribution parameters, improving hyper-parameter selection.

Findings

The new interpretation relates the transition point to noise in data.

It enables intuitive hyper-parameter tuning based on noise estimation.

Demonstrated effectiveness on object detection models.

Abstract

The Huber loss is a robust loss function used for a wide range of regression tasks. To utilize the Huber loss, a parameter that controls the transitions from a quadratic function to an absolute value function needs to be selected. We believe the standard probabilistic interpretation that relates the Huber loss to the Huber density fails to provide adequate intuition for identifying the transition point. As a result, a hyper-parameter search is often necessary to determine an appropriate value. In this work, we propose an alternative probabilistic interpretation of the Huber loss, which relates minimizing the loss to minimizing an upper-bound on the Kullback-Leibler divergence between Laplace distributions, where one distribution represents the noise in the ground-truth and the other represents the noise in the prediction. In addition, we show that the parameters of the Laplace distributions are directly related to the transition point of the Huber loss. We demonstrate, through a toy problem, that the optimal transition point of the Huber loss is closely related to the distribution of the noise in the ground-truth data. As a result, our interpretation provides an intuitive way to identify well-suited hyper-parameters by approximating the amount of noise in the data, which we demonstrate through a case study and experimentation on the Faster R-CNN and RetinaNet object detectors.