The Limits of Differential Privacy in Online Learning

TL;DR

This paper explores the fundamental limitations of differential privacy in online learning, showing that approximate DP is necessary for adaptive adversaries and that private learners inevitably make infinite mistakes.

Contribution

It demonstrates the separation between pure and approximate DP in online learning and proves that private online learners cannot have finite mistake bounds in most cases.

Findings

Approximate DP is essential against adaptive adversaries.

Private learners must make infinitely many mistakes.

Finite mistake bounds are possible without privacy constraints.

Abstract

Differential privacy (DP) is a formal notion that restricts the privacy leakage of an algorithm when running on sensitive data, in which privacy-utility trade-off is one of the central problems in private data analysis. In this work, we investigate the fundamental limits of differential privacy in online learning algorithms and present evidence that separates three types of constraints: no DP, pure DP, and approximate DP. We first describe a hypothesis class that is online learnable under approximate DP but not online learnable under pure DP under the adaptive adversarial setting. This indicates that approximate DP must be adopted when dealing with adaptive adversaries. We then prove that any private online learner must make an infinite number of mistakes for almost all hypothesis classes. This essentially generalizes previous results and shows a strong separation between private and non-private settings since a finite mistake bound is always attainable (as long as the class is online learnable) when there is no privacy requirement.