On the Generalization Error of Differentially Private Algorithms via Typicality

TL;DR

This paper provides sharper, explicit bounds on the generalization error of differentially private algorithms using information-theoretic measures and typicality arguments.

Contribution

It improves mutual information bounds and introduces new maximal leakage bounds for private algorithms, enhancing generalization error guarantees.

Findings

Sharper mutual information bounds for private algorithms.

New upper bounds on maximal leakage.

Explicit formulas translating information bounds into generalization guarantees.

Abstract

We study the generalization error of stochastic learning algorithms from an information-theoretic perspective, with a particular emphasis on deriving sharper bounds for differentially private algorithms. It is well known that the generalization error of stochastic learning algorithms can be bounded in terms of mutual information and maximal leakage, yielding in-expectation and high-probability guarantees, respectively. In this work, we further upper bound mutual information and maximal leakage by explicit, easily computable formulas, using typicality-based arguments and exploiting the stability properties of private algorithms. In the first part of the paper, we strictly improve the mutual-information bounds by Rodr\'iguez-G\'alvez et al. (IEEE Trans. Inf. Theory, 2021). In the second part, we derive new upper bounds on the maximal leakage of learning algorithms. In both cases, the resulting bounds on information measures translate directly into generalization error guarantees.