# Data-dependent PAC-Bayes priors via differential privacy

## TL;DR

This paper introduces a method to derive valid PAC-Bayes generalization bounds using data-dependent priors chosen via differential privacy techniques, demonstrating empirical benefits over traditional bounds.

## Contribution

It proposes a novel approach to obtain PAC-Bayes bounds with data-dependent priors through differential privacy, including practical mechanisms like SGLD for prior selection.

## Key findings

- Data-dependent bounds can be nonvacuous where others are vacuous.
- Differential privacy ensures valid PAC-Bayes bounds with data-dependent priors.
- Empirical results show improved generalization bounds in practice.

## Abstract

The Probably Approximately Correct (PAC) Bayes framework (McAllester, 1999) can incorporate knowledge about the learning algorithm and (data) distribution through the use of distribution-dependent priors, yielding tighter generalization bounds on data-dependent posteriors. Using this flexibility, however, is difficult, especially when the data distribution is presumed to be unknown. We show how an {\epsilon}-differentially private data-dependent prior yields a valid PAC-Bayes bound, and then show how non-private mechanisms for choosing priors can also yield generalization bounds. As an application of this result, we show that a Gaussian prior mean chosen via stochastic gradient Langevin dynamics (SGLD; Welling and Teh, 2011) leads to a valid PAC-Bayes bound given control of the 2-Wasserstein distance to an {\epsilon}-differentially private stationary distribution. We study our data-dependent bounds empirically, and show that they can be nonvacuous even when other distribution-dependent bounds are vacuous.