Tighter Generalization Bounds for Iterative Differentially Private Learning Algorithms

TL;DR

This paper establishes tighter generalization bounds for iterative differentially private learning algorithms, linking privacy preservation with generalization, and applies these bounds to deep learning scenarios like federated learning.

Contribution

It introduces new composition theorems for iterative algorithms and derives bounds that are tighter and independent of model size, advancing understanding of privacy and generalization.

Findings

Generalization bounds are tighter than previous work.

Bounds do not depend on model size, relevant for deep learning.

Applicable to stochastic gradient Langevin dynamics and federated learning.

Abstract

This paper studies the relationship between generalization and privacy preservation in iterative learning algorithms by two sequential steps. We first establish an alignment between generalization and privacy preservation for any learning algorithm. We prove that $(\varepsilon, \delta)$-differential privacy implies an on-average generalization bound for multi-database learning algorithms which further leads to a high-probability bound for any learning algorithm. This high-probability bound also implies a PAC-learnable guarantee for differentially private learning algorithms. We then investigate how the iterative nature shared by most learning algorithms influence privacy preservation and further generalization. Three composition theorems are proposed to approximate the differential privacy of any iterative algorithm through the differential privacy of its every iteration. By integrating the above two steps, we eventually deliver generalization bounds for iterative learning algorithms, which suggest one can simultaneously enhance privacy preservation and generalization. Our results are strictly tighter than the existing works. Particularly, our generalization bounds do not rely on the model size which is prohibitively large in deep learning. This sheds light to understanding the generalizability of deep learning. These results apply to a wide spectrum of learning algorithms. In this paper, we apply them to stochastic gradient Langevin dynamics and agnostic federated learning as examples.