Sample-efficient proper PAC learning with approximate differential privacy

TL;DR

This paper establishes a polynomial sample complexity for properly learning classes with finite Littlestone dimension under approximate differential privacy, improving previous exponential bounds and answering open questions.

Contribution

It provides the first polynomial sample complexity bound for proper private PAC learning of Littlestone classes, and introduces the concept of irreducibility in hypothesis classes.

Findings

Sample complexity is $ ilde O(d^6)$ for Littlestone dimension $d$.

Proper private learners exist with finite sample complexity, answering prior open questions.

Sanitization of hypothesis classes is polynomial in Littlestone and dual Littlestone dimensions.

Abstract

In this paper we prove that the sample complexity of properly learning a class of Littlestone dimension $d$ with approximate differential privacy is $\tilde O(d^6)$, ignoring privacy and accuracy parameters. This result answers a question of Bun et al. (FOCS 2020) by improving upon their upper bound of $2^{O(d)}$ on the sample complexity. Prior to our work, finiteness of the sample complexity for privately learning a class of finite Littlestone dimension was only known for improper private learners, and the fact that our learner is proper answers another question of Bun et al., which was also asked by Bousquet et al. (NeurIPS 2020). Using machinery developed by Bousquet et al., we then show that the sample complexity of sanitizing a binary hypothesis class is at most polynomial in its Littlestone dimension and dual Littlestone dimension. This implies that a class is sanitizable if and only if it has finite Littlestone dimension. An important ingredient of our proofs is a new property of binary hypothesis classes that we call irreducibility, which may be of independent interest.