Symmetrization of Loss Functions for Robust Training of Neural Networks in the Presence of Noisy Labels

TL;DR

This paper introduces a symmetrization method for loss functions to improve neural network robustness against noisy labels, proposing a unique convex multi-class symmetric loss and new loss functions with competitive performance.

Contribution

It develops a theoretical framework for symmetrizing multi-class loss functions, identifying the unique convex symmetric loss, and introduces new loss functions that interpolate between existing robust losses.

Findings

The multi-class unhinged loss is the unique convex symmetric loss under certain conditions.

Symmetrizing cross-entropy yields a linear extension of the unhinged loss.

Experiments show the proposed losses perform competitively on noisy-label benchmarks.

Abstract

Labeling a training set is often expensive and susceptible to errors, making the design of robust loss functions for label noise an important problem. The symmetry condition provides theoretical guarantees for robustness to such noise. In this work, we study a symmetrization method arising from the unique decomposition of any multi-class loss function into a symmetric component and a class-insensitive term. In particular, symmetrizing the cross-entropy loss leads to a linear multi-class extension of the unhinged loss. Unlike in the binary case, the multi-class version must have specific coefficients in order to satisfy the symmetry condition. Under suitable assumptions, we show that this multi-class unhinged loss is the unique convex multi-class symmetric loss. We also show that it has a fundamental local role: the linear approximation of any symmetric loss around score vectors with equal components is equivalent to the multi-class unhinged loss. We then introduce SGCE and alpha-MAE, two loss functions that interpolate between the multi-class unhinged loss and the Mean Absolute Error while allowing control of the beta-smoothness of the loss. Experiments on standard noisy-label benchmarks show competitive performance compared with existing robust loss functions.