Characterizing Online and Private Learnability under Distributional Constraints via Generalized Smoothness

TL;DR

This paper characterizes when online and private learning are feasible under distributional constraints, introducing generalized smoothness as a key condition for learnability and providing algorithms with near-optimal regret bounds.

Contribution

It offers a near complete characterization of distribution families allowing learnability via generalized smoothness and develops universal algorithms that adapt to these conditions.

Findings

Generalized smoothness characterizes learnability under distributional adversaries.

Universal algorithms achieve low regret without prior knowledge of the distribution family.

The results connect online learning, differential privacy, and distributional constraints.

Abstract

Understanding minimal assumptions that enable learning and generalization is perhaps the central question of learning theory. Several celebrated results in statistical learning theory, such as the VC theorem and Littlestone's characterization of online learnability, establish conditions on the hypothesis class that allow for learning under independent data and adversarial data, respectively. Building upon recent work bridging these extremes, we study sequential decision making under distributional adversaries that can adaptively choose data-generating distributions from a fixed family $U$ and ask when such problems are learnable with sample complexity that behaves like the favorable independent case. We provide a near complete characterization of families $U$ that admit learnability in terms of a notion known as generalized smoothness i.e. a distribution family admits VC-dimension-dependent regret bounds for every finite-VC hypothesis class if and only if it is generalized smooth. Further, we give universal algorithms that achieve low regret under any generalized smooth adversary without explicit knowledge of $U$. Finally, when $U$ is known, we provide refined bounds in terms of a combinatorial parameter, the fragmentation number, that captures how many disjoint regions can carry nontrivial mass under $U$. These results provide a nearly complete understanding of learnability under distributional adversaries. In addition, building upon the surprising connection between online learning and differential privacy, we show that the generalized smoothness also characterizes private learnability under distributional constraints.