Probing the Transition to Dataset-Level Privacy in ML Models Using an Output-Specific and Data-Resolved Privacy Profile

TL;DR

This paper introduces a novel privacy metric and profile to visualize and quantify data privacy in machine learning models trained with differential privacy, aiding in better privacy budget selection.

Contribution

It proposes a new coverage-based privacy metric and privacy profile to assess individual sample privacy and visualize the transition to indistinguishability in DP models.

Findings

The privacy profile effectively ranks individual sample privacy.

The transition to indistinguishability can be visualized using the privacy profile.

The approach aids in selecting appropriate privacy budgets for DP models.

Abstract

Differential privacy (DP) is the prevailing technique for protecting user data in machine learning models. However, deficits to this framework include a lack of clarity for selecting the privacy budget $\epsilon$ and a lack of quantification for the privacy leakage for a particular data row by a particular trained model. We make progress toward these limitations and a new perspective by which to visualize DP results by studying a privacy metric that quantifies the extent to which a model trained on a dataset using a DP mechanism is ``covered" by each of the distributions resulting from training on neighboring datasets. We connect this coverage metric to what has been established in the literature and use it to rank the privacy of individual samples from the training set in what we call a privacy profile. We additionally show that the privacy profile can be used to probe an observed transition to indistinguishability that takes place in the neighboring distributions as $\epsilon$ decreases, which we suggest is a tool that can enable the selection of $\epsilon$ by the ML practitioner wishing to make use of DP.