Private Selection with Heterogeneous Sensitivities

TL;DR

This paper investigates how to improve differentially private selection mechanisms by leveraging heterogeneity in candidate sensitivities, proposing adaptive methods that outperform standard approaches in various settings.

Contribution

It introduces a correlation-based heuristic and a combined mechanism that adaptively chooses between existing DP selection algorithms, enhancing performance across diverse scenarios.

Findings

Combined GEM outperforms standard mechanisms in polarized settings.

Heterogeneity in sensitivities can be exploited for better DP selection.

No single mechanism is best in all cases, adaptive methods are advantageous.

Abstract

Differentially private (DP) selection involves choosing a high-scoring candidate from a finite candidate pool, where each score depends on a sensitive dataset. This problem arises naturally in a variety of contexts including model selection, hypothesis testing, and within many DP algorithms. Classical methods, such as Report Noisy Max (RNM), assume all candidates' scores are equally sensitive to changes in a single individual's data, but this often isn't the case. To address this, algorithms like the Generalised Exponential Mechanism (GEM) leverage variability in candidate sensitivities. However, we observe that while these algorithms can outperform RNM in some situations, they may underperform in others - they can even perform worse than random selection. In this work, we explore how the distribution of scores and sensitivities impacts DP selection mechanisms. In all settings we study, we find that there exists a mechanism that utilises heterogeneity in the candidate sensitivities that outperforms standard mechanisms like RNM. However, no single mechanism uniformly outperforms RNM. We propose using the correlation between the scores and sensitivities as the basis for deciding which DP selection mechanism to use. Further, we design a slight variant of GEM, modified GEM that generally performs well whenever GEM performs poorly. Relying on the correlation heuristic we propose combined GEM, which adaptively chooses between GEM and modified GEM and outperforms both in polarised settings.