Distributed, communication-efficient, and differentially private estimation of KL divergence

TL;DR

This paper introduces new algorithms for estimating the KL divergence between distributions in federated settings that are both communication-efficient and differentially private, enabling privacy-preserving data drift analysis.

Contribution

The work presents novel algorithms for private KL divergence estimation in federated models, with theoretical analysis and empirical validation of their effectiveness.

Findings

Private estimators achieve accuracy comparable to non-private baselines.

Algorithms are optimized for different trust and privacy settings.

Empirical results demonstrate practical applicability in real-world tasks.

Abstract

A key task in managing distributed, sensitive data is to measure the extent to which a distribution changes. Understanding this drift can effectively support a variety of federated learning and analytics tasks. However, in many practical settings sharing such information can be undesirable (e.g., for privacy concerns) or infeasible (e.g., for high communication costs). In this work, we describe novel algorithmic approaches for estimating the KL divergence of data across federated models of computation, under differential privacy. We analyze their theoretical properties and present an empirical study of their performance. We explore parameter settings that optimize the accuracy of the algorithm catering to each of the settings; these provide sub-variations that are applicable to real-world tasks, addressing different context- and application-specific trust level requirements. Our experimental results confirm that our private estimators achieve accuracy comparable to a baseline algorithm without differential privacy guarantees.