Cauchy Loss Function: Robustness Under Gaussian and Cauchy Noise

TL;DR

This paper evaluates the robustness of the Cauchy loss function (CLF) against Gaussian and Cauchy noise, demonstrating its potential advantages over MSE in handling outliers in neural network training.

Contribution

It provides an empirical comparison of CLF and MSE, highlighting CLF's robustness and generalizability in noisy data scenarios.

Findings

CLF performs comparably or better than MSE in noisy conditions.

CLF shows robustness to outliers in both handcrafted and real-world datasets.

MSE may underperform in the presence of outliers.

Abstract

In supervised machine learning, the choice of loss function implicitly assumes a particular noise distribution over the data. For example, the frequently used mean squared error (MSE) loss assumes a Gaussian noise distribution. The choice of loss function during training and testing affects the performance of artificial neural networks (ANNs). It is known that MSE may yield substandard performance in the presence of outliers. The Cauchy loss function (CLF) assumes a Cauchy noise distribution, and is therefore potentially better suited for data with outliers. This papers aims to determine the extent of robustness and generalisability of the CLF as compared to MSE. CLF and MSE are assessed on a few handcrafted regression problems, and a real-world regression problem with artificially simulated outliers, in the context of ANN training. CLF yielded results that were either comparable to or better than the results yielded by MSE, with a few notable exceptions.