Noise Tolerance under Risk Minimization

TL;DR

This paper investigates how different risk minimization methods perform under noisy data, revealing that 0-1 loss offers strong noise tolerance, unlike squared error or other loss functions, which are less robust.

Contribution

It provides a theoretical analysis of noise tolerance in risk minimization, highlighting the superior robustness of 0-1 loss in noisy environments.

Findings

0-1 loss risk minimization is highly noise tolerant

Squared error loss is only tolerant to uniform noise

Other loss functions lack noise tolerance

Abstract

In this paper we explore noise tolerant learning of classifiers. We formulate the problem as follows. We assume that there is an ${\bf unobservable}$ training set which is noise-free. The actual training set given to the learning algorithm is obtained from this ideal data set by corrupting the class label of each example. The probability that the class label of an example is corrupted is a function of the feature vector of the example. This would account for most kinds of noisy data one encounters in practice. We say that a learning method is noise tolerant if the classifiers learnt with the ideal noise-free data and with noisy data, both have the same classification accuracy on the noise-free data. In this paper we analyze the noise tolerance properties of risk minimization (under different loss functions), which is a generic method for learning classifiers. We show that risk minimization under 0-1 loss function has impressive noise tolerance properties and that under squared error loss is tolerant only to uniform noise; risk minimization under other loss functions is not noise tolerant. We conclude the paper with some discussion on implications of these theoretical results.