Littlestone Classes are Privately Online Learnable

TL;DR

This paper demonstrates that Littlestone classes, characterized by their online learnability, can be privately learned in an online setting with regret bounds comparable to non-private algorithms, thus strengthening the connection between online and private learning.

Contribution

It provides the first non-trivial regret bounds for privately learning Littlestone classes in the realizable setting, showing direct privatization of online algorithms.

Findings

For classes with constant Littlestone dimension, private learners achieve $O( ext{log } T)$ mistakes.

The mistake bound extends to general Littlestone dimension with a doubly-exponential factor.

An adaptive setting with $O( ext{sqrt } T)$ regret is also discussed.

Abstract

We consider the problem of online classification under a privacy constraint. In this setting a learner observes sequentially a stream of labelled examples $(x_t, y_t)$, for $1 \leq t \leq T$, and returns at each iteration $t$ a hypothesis $h_t$ which is used to predict the label of each new example $x_t$. The learner's performance is measured by her regret against a known hypothesis class $\mathcal{H}$. We require that the algorithm satisfies the following privacy constraint: the sequence $h_1, \ldots, h_T$ of hypotheses output by the algorithm needs to be an $(\epsilon, \delta)$-differentially private function of the whole input sequence $(x_1, y_1), \ldots, (x_T, y_T)$. We provide the first non-trivial regret bound for the realizable setting. Specifically, we show that if the class $\mathcal{H}$ has constant Littlestone dimension then, given an oblivious sequence of labelled examples, there is a private learner that makes in expectation at most $O(\log T)$ mistakes -- comparable to the optimal mistake bound in the non-private case, up to a logarithmic factor. Moreover, for general values of the Littlestone dimension $d$, the same mistake bound holds but with a doubly-exponential in $d$ factor. A recent line of work has demonstrated a strong connection between classes that are online learnable and those that are differentially-private learnable. Our results strengthen this connection and show that an online learning algorithm can in fact be directly privatized (in the realizable setting). We also discuss an adaptive setting and provide a sublinear regret bound of $O(\sqrt{T})$.