Mixing between the Cross Entropy and the Expectation Loss Terms

TL;DR

This paper proposes a novel training strategy that combines cross entropy and expectation loss, shifting focus during training to improve accuracy across various domains.

Contribution

It introduces a dynamic loss mixing method that transitions from cross entropy to expectation loss during training, enhancing classification performance.

Findings

Improved accuracy across vision, NLP, tabular, and sequence tasks.

Effective in focusing on misclassified samples with higher probabilities.

Outperforms traditional training methods in diverse domains.

Abstract

The cross entropy loss is widely used due to its effectiveness and solid theoretical grounding. However, as training progresses, the loss tends to focus on hard to classify samples, which may prevent the network from obtaining gains in performance. While most work in the field suggest ways to classify hard negatives, we suggest to strategically leave hard negatives behind, in order to focus on misclassified samples with higher probabilities. We show that adding to the optimization goal the expectation loss, which is a better approximation of the zero-one loss, helps the network to achieve better accuracy. We, therefore, propose to shift between the two losses during training, focusing more on the expectation loss gradually during the later stages of training. Our experiments show that the new training protocol improves performance across a diverse set of classification domains, including computer vision, natural language processing, tabular data, and sequences. Our code and scripts are available at supplementary.