Robust Bi-Tempered Logistic Loss Based on Bregman Divergences

TL;DR

This paper proposes a bi-tempered logistic loss function for neural networks, introducing temperature parameters into the softmax and log loss to improve robustness to noise, based on Bregman divergences.

Contribution

It introduces a novel bi-tempered logistic loss with two temperature parameters, enhancing robustness and outperforming previous divergence-based methods.

Findings

Improved robustness to noisy data

Effective on large datasets

Outperforms Tsallis divergence-based methods

Abstract

We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization. Similarly, the logarithm in the log loss we use for training is replaced by a low temperature logarithm. By tuning the two temperatures we create loss functions that are non-convex already in the single layer case. When replacing the last layer of the neural nets by our bi-temperature generalization of logistic loss, the training becomes more robust to noise. We visualize the effect of tuning the two temperatures in a simple setting and show the efficacy of our method on large data sets. Our methodology is based on Bregman divergences and is superior to a related two-temperature method using the Tsallis divergence.