Cut your Losses with Squentropy

TL;DR

This paper introduces the squentropy loss, a new combined loss function for neural classification that improves accuracy, calibration, and stability over traditional cross-entropy and square losses, without additional tuning.

Contribution

The paper proposes the squentropy loss, a novel combination of cross-entropy and square loss, which enhances classification performance and calibration without extra parameter tuning.

Findings

Squentropy outperforms cross-entropy and square loss in accuracy.

It provides better model calibration than alternatives.

It has less variance across different initializations.

Abstract

Nearly all practical neural models for classification are trained using cross-entropy loss. Yet this ubiquitous choice is supported by little theoretical or empirical evidence. Recent work (Hui & Belkin, 2020) suggests that training using the (rescaled) square loss is often superior in terms of the classification accuracy. In this paper we propose the "squentropy" loss, which is the sum of two terms: the cross-entropy loss and the average square loss over the incorrect classes. We provide an extensive set of experiments on multi-class classification problems showing that the squentropy loss outperforms both the pure cross entropy and rescaled square losses in terms of the classification accuracy. We also demonstrate that it provides significantly better model calibration than either of these alternative losses and, furthermore, has less variance with respect to the random initialization. Additionally, in contrast to the square loss, squentropy loss can typically be trained using exactly the same optimization parameters, including the learning rate, as the standard cross-entropy loss, making it a true "plug-and-play" replacement. Finally, unlike the rescaled square loss, multiclass squentropy contains no parameters that need to be adjusted.