Private PAC learning implies finite Littlestone dimension

TL;DR

This paper proves that private PAC learning algorithms require a minimum number of examples related to the Littlestone dimension, showing limitations for private learning of certain classes like thresholds.

Contribution

It establishes a lower bound on the sample complexity for private PAC learning based on Littlestone dimension, resolving an open question for threshold classes.

Findings

Private PAC learning requires at least logarithmic iterated Littlestone dimension examples.

Threshold classes over natural numbers cannot be learned privately.

Open question remains whether all finite Littlestone dimension classes are privately learnable.

Abstract

We show that every approximately differentially private learning algorithm (possibly improper) for a class $H$ with Littlestone dimension~$d$ requires $\Omega\bigl(\log^*(d)\bigr)$ examples. As a corollary it follows that the class of thresholds over $\mathbb{N}$ can not be learned in a private manner; this resolves open question due to [Bun et al., 2015, Feldman and Xiao, 2015]. We leave as an open question whether every class with a finite Littlestone dimension can be learned by an approximately differentially private algorithm.