Normalized Loss Functions for Deep Learning with Noisy Labels

TL;DR

This paper introduces a framework called Active Passive Loss (APL) that creates robust loss functions for deep neural networks, significantly improving training accuracy under high label noise by combining two mutually boosting robust losses.

Contribution

The paper presents a theoretical normalization technique to make any loss robust to noisy labels and introduces the APL framework that combines robust losses to prevent underfitting.

Findings

APL-based loss functions outperform state-of-the-art methods under high noise rates.

Normalization makes any loss function robust to noisy labels.

Robust loss functions alone may cause underfitting, which APL addresses.

Abstract

Robust loss functions are essential for training accurate deep neural networks (DNNs) in the presence of noisy (incorrect) labels. It has been shown that the commonly used Cross Entropy (CE) loss is not robust to noisy labels. Whilst new loss functions have been designed, they are only partially robust. In this paper, we theoretically show by applying a simple normalization that: any loss can be made robust to noisy labels. However, in practice, simply being robust is not sufficient for a loss function to train accurate DNNs. By investigating several robust loss functions, we find that they suffer from a problem of underfitting. To address this, we propose a framework to build robust loss functions called Active Passive Loss (APL). APL combines two robust loss functions that mutually boost each other. Experiments on benchmark datasets demonstrate that the family of new loss functions created by our APL framework can consistently outperform state-of-the-art methods by large margins, especially under large noise rates such as 60% or 80% incorrect labels.