Distribution Learnability and Robustness

TL;DR

This paper investigates the differences between learnability and robust learnability of probability distributions, revealing that realizable learnability does not imply agnostic learnability and that robustness depends on the type of data corruption.

Contribution

It demonstrates that realizable learnability does not guarantee agnostic learnability for distribution classes and analyzes robustness against different types of data corruption.

Findings

Realizable learnability does not imply agnostic learnability.

Robust learnability holds against additive corruption but not subtractive corruption.

Implications for compression schemes and differential privacy are discussed.

Abstract

We examine the relationship between learnability and robust (or agnostic) learnability for the problem of distribution learning. We show that, contrary to other learning settings (e.g., PAC learning of function classes), realizable learnability of a class of probability distributions does not imply its agnostic learnability. We go on to examine what type of data corruption can disrupt the learnability of a distribution class and what is such learnability robust against. We show that realizable learnability of a class of distributions implies its robust learnability with respect to only additive corruption, but not against subtractive corruption. We also explore related implications in the context of compression schemes and differentially private learnability.