Robust Learnability of Sample-Compressible Distributions under Noisy or Adversarial Perturbations

TL;DR

This paper demonstrates that sample-compressible distribution families remain learnable under noisy or adversarial sample perturbations, providing new bounds and resolving open problems in high-dimensional mixture model learning.

Contribution

It establishes necessary and sufficient conditions for learnability of sample-compressible families under perturbations, extending prior work to noisy and adversarial settings.

Findings

Sample compressibility ensures learnability under noise and adversarial corruption.

New sample complexity bounds for high-dimensional mixture models under perturbations.

Resolved open problems in learning Gaussian mixtures with adversarial samples.

Abstract

Learning distribution families over $\mathbb{R}^d$ is a fundamental problem in unsupervised learning and statistics. A central question in this setting is whether a given family of distributions possesses sufficient structure to be (at least) information-theoretically learnable and, if so, to characterize its sample complexity. In 2018, Ashtiani et al. reframed \emph{sample compressibility}, originally due to Littlestone and Warmuth (1986), as a structural property of distribution classes, proving that it guarantees PAC-learnability. This discovery subsequently enabled a series of recent advancements in deriving nearly tight sample complexity bounds for various high-dimensional open problems. It has been further conjectured that the converse also holds: every learnable class admits a tight sample compression scheme. In this work, we establish that sample compressible families remain learnable even from perturbed samples, subject to a set of necessary and sufficient conditions. We analyze two models of data perturbation: (i) an additive independent noise model, and (ii) an adversarial corruption model, where an adversary manipulates a limited subset of the samples unknown to the learner. Our results are general and rely on as minimal assumptions as possible. We develop a perturbation-quantization framework that interfaces naturally with the compression scheme and leads to sample complexity bounds that scale gracefully with the noise level and corruption budget. As concrete applications, we establish new sample complexity bounds for learning finite mixtures of high-dimensional uniform distributions under both noise and adversarial perturbations, as well as for learning Gaussian mixture models from adversarially corrupted samples, resolving two open problems in the literature.