Is nasty noise actually harder than malicious noise?

TL;DR

This paper investigates the difficulty of learning Boolean functions under two adversarial noise models, revealing a surprising equivalence in distribution-independent settings and a significant separation in fixed-distribution scenarios.

Contribution

It establishes a strong equivalence between malicious and nasty noise in distribution-independent learning, and demonstrates an arbitrarily large separation in fixed-distribution settings, introducing the ICE class of algorithms.

Findings

Distribution-independent learning: malicious and nasty noise are equivalent.

Fixed-distribution setting: large separation under cryptographic assumptions.

ICE algorithms: malicious and nasty noise are equivalent up to a factor of two.

Abstract

We consider the relative abilities and limitations of computationally efficient algorithms for learning in the presence of noise, under two well-studied and challenging adversarial noise models for learning Boolean functions: malicious noise, in which an adversary can arbitrarily corrupt a random subset of examples given to the learner; and nasty noise, in which an adversary can arbitrarily corrupt an adversarially chosen subset of examples given to the learner. We consider both the distribution-independent and fixed-distribution settings. Our main results highlight a dramatic difference between these two settings: For distribution-independent learning, we prove a strong equivalence between the two noise models: If a class ${\cal C}$ of functions is efficiently learnable in the presence of $\eta$-rate malicious noise, then it is also efficiently learnable in the presence of $\eta$-rate nasty noise. In sharp contrast, for the fixed-distribution setting we show an arbitrarily large separation: Under a standard cryptographic assumption, for any arbitrarily large value $r$ there exists a concept class for which there is a ratio of $r$ between the rate $\eta_{malicious}$ of malicious noise that polynomial-time learning algorithms can tolerate, versus the rate $\eta_{nasty}$ of nasty noise that such learning algorithms can tolerate. To offset the negative result for the fixed-distribution setting, we define a broad and natural class of algorithms, namely those that ignore contradictory examples (ICE). We show that for these algorithms, malicious noise and nasty noise are equivalent up to a factor of two in the noise rate: Any efficient ICE learner that succeeds with $\eta$-rate malicious noise can be converted to an efficient learner that succeeds with $\eta/2$-rate nasty noise. We further show that the above factor of two is necessary, again under a standard cryptographic assumption.