Adaptive Data Analysis with Correlated Observations

TL;DR

This paper explores adaptive data analysis in the context of correlated observations, extending existing methods to dependent data and establishing conditions under which privacy guarantees still ensure generalization.

Contribution

It introduces the concept of Gibbs-dependence to quantify correlations and demonstrates that differential privacy can still provide generalization guarantees in dependent data settings.

Findings

Differential privacy guarantees generalization under Gibbs-dependence.

Extension of transcript-compression techniques to non-iid data.

Identification of limitations through a tight negative example.

Abstract

The vast majority of the work on adaptive data analysis focuses on the case where the samples in the dataset are independent. Several approaches and tools have been successfully applied in this context, such as differential privacy, max-information, compression arguments, and more. The situation is far less well-understood without the independence assumption. We embark on a systematic study of the possibilities of adaptive data analysis with correlated observations. First, we show that, in some cases, differential privacy guarantees generalization even when there are dependencies within the sample, which we quantify using a notion we call Gibbs-dependence. We complement this result with a tight negative example. Second, we show that the connection between transcript-compression and adaptive data analysis can be extended to the non-iid setting.