Curriculum Loss: Robust Learning and Generalization against Label Corruption

TL;DR

This paper introduces curriculum loss (CL), a new efficient loss function that enhances robustness and generalization of deep neural networks against label corruption by adaptively selecting training samples.

Contribution

The paper proposes curriculum loss (CL), a novel surrogate for 0-1 loss that is computationally efficient and improves robustness by integrating curriculum sample selection.

Findings

CL is a tighter upper bound of 0-1 loss than traditional surrogates.

CL adaptively selects samples, enhancing robustness against label noise.

Experimental results show improved robustness on benchmark datasets.

Abstract

Deep neural networks (DNNs) have great expressive power, which can even memorize samples with wrong labels. It is vitally important to reiterate robustness and generalization in DNNs against label corruption. To this end, this paper studies the 0-1 loss, which has a monotonic relationship with an empirical adversary (reweighted) risk~\citep{hu2016does}. Although the 0-1 loss has some robust properties, it is difficult to optimize. To efficiently optimize the 0-1 loss while keeping its robust properties, we propose a very simple and efficient loss, i.e. curriculum loss (CL). Our CL is a tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses. Moreover, CL can adaptively select samples for model training. As a result, our loss can be deemed as a novel perspective of curriculum sample selection strategy, which bridges a connection between curriculum learning and robust learning. Experimental results on benchmark datasets validate the robustness of the proposed loss.