SoK: Differential Privacies

TL;DR

This paper provides a comprehensive systematic taxonomy of the various variants and extensions of differential privacy, analyzing their properties, relationships, and compositional characteristics.

Contribution

It introduces a structured taxonomy of differential privacy variants, categorizes them, and summarizes their relative strengths and properties with new and existing proofs.

Findings

Seven categories of differential privacy variants identified

Partial ordering of the strength of privacy notions established

Properties like composition and convexity analyzed for each variant

Abstract

Shortly after it was first introduced in 2006, differential privacy became the flagship data privacy definition. Since then, numerous variants and extensions were proposed to adapt it to different scenarios and attacker models. In this work, we propose a systematic taxonomy of these variants and extensions. We list all data privacy definitions based on differential privacy, and partition them into seven categories, depending on which aspect of the original definition is modified. These categories act like dimensions: variants from the same category cannot be combined, but variants from different categories can be combined to form new definitions. We also establish a partial ordering of relative strength between these notions by summarizing existing results. Furthermore, we list which of these definitions satisfy some desirable properties, like composition, post-processing, and convexity by either providing a novel proof or collecting existing ones.

Tables

Table 1. Table 1: Notations used in this paper.

Notation	Description
$𝒯$	Set of possible records
$t\in 𝒯$	A possible record
$𝒟={𝒯}^{\ast }$	Set of possible datasets (sequences of records)
$D\in 𝒟$	Dataset (we also use $D^{\prime },D_1$, $D_2$, …)
$D\left(i\right)$	$i$-th record of the dataset ($i\le \left|D\right|$)
$D_{-i}$	Dataset $D$, with its $i$-th record removed
$𝒪$	Set of possible outputs of privacy mechanisms
$S\subseteq 𝒪$	Subset of possible outputs
$O\in 𝒪$	Output of the privacy mechanism
$ℳ:𝒟\to 𝒪$	Privacy mechanism (probabilistic)
$ℳ\left(D\right)$	The distribution (or an instance thereof) of the outputs of $ℳ$ given input $D$
$d_{𝒟}:𝒟\times 𝒟\to {ℝ}_0^{+}$	Distance function between datasets
$\varphi \subseteq 𝒟$	Predicate on datasets
$\Phi$	Family of sensitive predicates on datasets
$\pi$	Probability distribution on $𝒯$
$ℬ$	Set of possible background knowledges
$B\in ℬ$	Background knowledge (we also use $\hat{B}$)
$\Theta$	Family of probability distributions on $𝒟$, on $𝒟\times ℬ$
$\theta \in \Theta$	Probability distribution on $𝒟$, on $𝒟\times ℬ$
$ℳ{\left(D\right)}_{|D\sim \theta ,\varphi }$	Distribution of outputs of $ℳ$ given an input drawn from $\theta$, conditioned on $\varphi$
$\Omega$	Probabilistic polynomial-time Turing machine, called distinguisher

Table 2. Table 3: Summary of variants/extensions of DP representing the main options in each combination of dimensions.

Name & References	Dimension	Axioms		Cp.	Relations
		P.P.	Cv.
		10	10	10
$(\epsilon ,\delta )$-approximate DP [DKM+06]	Q	$✓$10	$✓$10	$✓$10	$(\epsilon ,\delta )$-DP ${\supset }^{\prec }$ $\epsilon$-DP
$(\epsilon ,\delta )$-probabilistic DP [Mei18]	Q	✗10	✗10	$✓$10	$(\epsilon ,\delta )$-DP $\sim$ $(\epsilon ,\delta )$-ProDP
$(\epsilon ,{\delta }_a,{\delta }_p)$-Relaxed DP [ZQZ+15]	Q	✗10	✗10	$✓$10	$(\epsilon ,{\delta }_p)$-ProDP ${\subset }^{\succ }$ $(\epsilon ,{\delta }_a,{\delta }_p)$-RelDP
					$(\epsilon ,{\delta }_a)$-DP ${\subset }^{\succ }$ $(\epsilon ,{\delta }_a,{\delta }_p)$-RelDP
$\epsilon$-Kullback-Leiber Pr [CY16]	Q	$✓$10	$✓$10	$✓$10	$(\epsilon ,\delta )$-DP $\prec$ $\epsilon$-KLPr $\prec$ $\epsilon$-DP
$(\alpha ,\epsilon )$-Rényi DP [Mir17]	Q	$✓$10	$✓$10	$✓$10	$\epsilon$-KLPr ${\subset }^{\prec }$$(\alpha ,\epsilon )$-RenyiDP ${\supset }^{\prec }$ $\epsilon$-DP
$(\mu ,\tau )$-mean concentrated DP [DR16]	Q	✗10	?	$✓$10	$(\epsilon ,\delta )$-DP $\prec$ $(\mu ,\tau )$-mCoDP $\prec$ $\epsilon$-DP
$(\xi ,\rho )$-zero concentrated DP [BS16]	Q	$✓$10	$✓$10	$✓$10	$(\xi ,\rho )$-zCoDP $\sim$ $(\mu ,\tau )$-mCoDP
$(\xi ,\rho ,\delta )$-approximate CoDP [BS16]	Q	✗10	?	$✓$10	$(\epsilon ,\delta )$-DP $\succ$ $(\xi ,\rho ,\delta )$-ACoDP ${\supset }^{\prec }$ $(\xi ,\rho )$-zCoDP
$(\xi ,\rho ,\omega )$-bounded CoDP [BS16]	Q	$✓$10	$✓$10	$✓$10	$(\xi ,\rho ,\omega )$-BCoDP ${\supset }^{\prec }$ $(\xi ,\rho )$-zCoDP
$(\eta ,\tau )$-truncated CoDP [Col16]	Q	$✓$10	$✓$10	$✓$10	$(\eta ,\tau )$-TCoDP[Col16] $\sim$ $\epsilon$-DP
$(\rho ,\omega )$-truncated CoDP [BDRS18]	Q	$✓$10	$✓$10	$✓$10	$(\rho ,\omega )$-TCoDP[BDRS18] ${\subset }^{\prec }$ $(\xi ,\rho ,\omega )$-bCoDP
$(f,\epsilon )$-divergence DP [BD14]	Q	$✓$10	$✓$10	?	$(f,\epsilon )$-DivDP $\supset$ most def. in Q
$(f_k,\epsilon )$-divergence DP [DR18]	Q	$✓$10	$✓$10	?	$(f_k,\epsilon )$-kDivDP $\subset$ $(f,\epsilon )$-DivDP
$(H,f,\epsilon )$-capacity bounded DP [CIM19]	Q	$✓$10	$✓$10	?	$(H,f,\epsilon )$-CBDP ${\subset }^{\prec }$$(f,\epsilon )$-DivDP
$\epsilon$-unbounded DP [KM11]	N	$✓$10	$✓$10	$✓$10	$\epsilon$-DP $\sim$ $\epsilon$-uBoDP ${\subset }^{\sim }$ $(c,\epsilon )$-GrDP
$\epsilon$-bounded DP [KM11]	N	$✓$10	$✓$10	$✓$10	$\epsilon$-BoDP $\prec$ $\epsilon$-DP
$(P,\epsilon )$-one-sided DP [KDH+20]	N	$✓$10	$✓$10	$✓$10	$(P,\epsilon )$-OnSDP ${\supset }^{\prec }$ $\epsilon$-BoDP
$(P,\epsilon )$-asymetric DP [TCY21]	N	$✓$10	$✓$10	$✓$10	$(P,\epsilon )$-AsDP ${\supset }^{\prec }$ $\epsilon$-uBoDP
$\epsilon$-client DP [MRTZ18]	N	$✓$10	$✓$10	$✓$10	$\epsilon$-ClDP $\succ$ $\epsilon$-DP
$\epsilon$-element DP [ADJ19]	N	$✓$10	$✓$10	$✓$10	$\epsilon$-ClDP $\succ$ $\epsilon$-ElDP $\succ$ $\epsilon$-DP
$(c,\epsilon )$-group DP [Dwo08]	N	$✓$10	$✓$10	$✓$10	$(c,\epsilon )$-GrDP ${\supset }^{\sim }$ $\epsilon$-DP
$(R,c,\epsilon )$-dependent DP [LCM16]	N	$✓$10	$✓$10	$✓$10	$(R,c,\epsilon )$-DepDP $\supset$ $(c,\epsilon )$-GrDP
$(𝒜,\epsilon )$-bayesian DP [YSN15]	N	$✓$10	$✓$10	$✓$10	$(𝒜,\epsilon )$-BayDP[YSN15] $\supset$ $(R,c,\epsilon )$-DepDP
$(𝒜,\epsilon )$-correlated DP [WDN17, WWK+17]	N	$✓$10	$✓$10	$✓$10	$(𝒜,\epsilon )$-CorDP $\subset$ $(𝒜,\epsilon )$-BayDP[YSN15]
$(𝒜,\epsilon )$-prior DP [LRYY19]	N	$✓$10	$✓$10	$✓$10	$(𝒜,\epsilon )$-PriDP $\supset$ $(𝒜,\epsilon )$-BayDP[YSN15]
$\epsilon$-free lunch Pr [KM11]	N	$✓$10	$✓$10	$✓$10	$\epsilon$-FLPr $\succ$ all def. in N
$(D,\epsilon )$-individual DP [SCDFSM17]	N	$✓$10	$✓$10	$✓$10	$(D,\epsilon )$-IndDP $\prec$ $\epsilon$-DP
$(D,t,\epsilon )$-per-instance DP [RW21]	N	$✓$10	$✓$10	$✓$10	$(D,t,\epsilon )$-PIDP $\prec$ $(D,\epsilon )$-IndDP
$(ℛ,\epsilon )$-generic DP [KM11, FC14]	N	$✓$10	$✓$10	$✓$10	$(ℛ,\epsilon )$-GcDP[KM11] $\supset$ most def. in N
$(d,\Delta ,\epsilon )$-constrained DP [ZLW09]	N	$✓$10	$✓$10	$✓$10	$(d,\Delta ,\epsilon )$-ConsDP $\sim$ $(ℛ,\epsilon )$-GcDP
$(d,\Delta ,S_{𝒟},\epsilon )$-distributional Pr [ZLW09]	N	$✓$10	$✓$10	$✓$10	$(d,\Delta ,S_{𝒟},\epsilon )$-DlPr[ZLW09] $\subset$ $(ℛ,\epsilon )$-GcDP
$(f,\epsilon )$-sensitivity-induced DP [RA17]	N	$✓$10	$✓$10	$✓$10	$\epsilon$-SIDP $\subset$ $(ℛ,\epsilon )$-GcDP
$({ℐ}_Q,\epsilon )$-induced-neighbors DP [KM11]	N	$✓$10	$✓$10	$✓$10	$({ℐ}_Q,\epsilon )$-INDP $\subset$ $(ℛ,\epsilon )$-GcDP
$(G,{ℐ}_Q,\epsilon )$-blowfish Pr [HMD14]	N	$✓$10	$✓$10	$✓$10	$(G,{ℐ}_Q,\epsilon )$-BFPr $\subset$ $(ℛ,\epsilon )$-GcDP
$\Psi$-personalized DP [JYC15]	V	$✓$10	$✓$10	$✓$10	$\Psi$-PerDP $\supset$ $\epsilon$-DP
$\Psi$-tailored DP [LP15]	V	$✓$10	$✓$10	$✓$10	$\Psi$-TaiDP $\supset$ $\Psi$-PerDP
$(\Psi ,\overline{\epsilon })$-input-discriminative DP [GLXC20]	V	$✓$10	$✓$10	$✓$10	$\Psi$-IDDP $\supset$ $\Psi$-PerDP
$\epsilon (\cdot )$-outlier Pr [LP15]	V	$✓$10	$✓$10	$✓$10	$\epsilon (\cdot )$-OutPr $\subset$ $\Psi$-TaiDP
$(\epsilon ,p,r)$-Pareto DP [LP15]	V	$✓$10	$✓$10	$✓$10	$(\epsilon ,p,r)$-ParDP $\subset$ $\Psi$-TaiDP
$(\pi ,\gamma ,\epsilon )$-random DP [HWR13]	V	?	✗10	$✓$10	$(\pi ,\gamma ,\epsilon )$-RanDP ${\supset }^{\prec }$ $\epsilon$-DP
$d_{𝒟}$-Pr [CABP13]	N,V	$✓$10	$✓$10	$✓$10	$\epsilon$-DP $\subset$ $d_{𝒟}$-Pr
$(d_{𝒟},\epsilon )$-smooth DP [BD14]	N,V	$✓$10	$✓$10	$✓$10	$(d_{𝒟},\epsilon )$-SmDP $\sim$ $d_{𝒟}$-Pr
$(\epsilon ,\gamma )$-distributional Pr [ZLW09, Rot10]	N,V	?	?	?	$\epsilon$-FLPr $\subset$ $(\epsilon ,\gamma )$-DlPr[Rot10, BLR13]
$(\pi ,\epsilon ,\delta )$-weak Bayesian DP [TF20]	Q,V	$✓$10	?	$✓$10	$(\epsilon ,\delta )$-DP $\succ$ $(\pi ,\epsilon ,\delta )$-WBDP $\prec$ $(\pi ,\gamma ,\epsilon )$-RanDP
$(\Theta ,\epsilon )$-on average KL Pr [WLF16]	Q,V	$✓$10	?	?10	$\epsilon$-KLPr $\succ$ $(\Theta ,\epsilon )$-avgKLPr $\prec$ $(\pi ,\gamma ,\epsilon )$-RanDP
$(\pi ,\epsilon ,\delta )$-Bayesian DP [TF20]	Q,V	$✓$10	✗10	$✓$10	$(\epsilon ,\delta )$-ProDP $\succ$ $(\pi ,\epsilon ,\delta )$-BayDP[TF20]
					$(\pi ,\gamma ,\epsilon )$-RanDP $\succ$ $(\pi ,\epsilon ,\delta )$-BayDP[TF20]
$(d_{𝒟},\epsilon ,\delta )$-pseudo-metric DP [DNM+13]	Q,N,V	?	?	$✓$10	$(\epsilon ,\delta )$-DP $\subset$ $(d_{𝒟},\epsilon ,\delta )$-PsDP ${\supset }^{\prec }$ $d_{𝒟}$-Pr
$(f,d,\epsilon )$-extended divergence DP [KM19a]	Q,N,V	$✓$10	$✓$10	?	$d_{𝒟}$-Pr $\subset$ $(f,d,\epsilon )$-EDivDP $\supset$ $(f,\epsilon )$-Div DP
$(ℛ,M)$-generic DP [KL10]	Q,N,V	$✓$10	$✓$10	?	$(ℛ,M)$-GcDP[KL10] $\supset$ $(\epsilon ,\delta )$-DP
$(ℛ,q)$-abstract DP [KL10]	Q,N,V	✗10	✗10	?	$(ℛ,q)$-AbsDP $\supset$ $(ℛ,M)$-GcDP[KL10]
$(\Theta ,\epsilon )$-noiseless Pr [BBG+11]	B	$✓$10	$✓$10	✗10	$\epsilon$-DP ${\supset }^{\prec }$ $(\Theta ,\epsilon )$-NPr
$(\Theta ,\epsilon )$-causal DP [DMKB19]	B	$✓$10	$✓$10	✗10	$(\Theta ,\epsilon )$-CausDP ${\supset }^{\prec }$ $\epsilon$-DP
$(\beta ,\epsilon )$-DP under sampling [LQS12]	Q,B	$✓$10	$✓$10	✗10	$(\Theta ,\epsilon )$-NPr $\supset$ $(\beta ,\epsilon )$-SamDP ${\supset }^{\prec }$ $\epsilon$-DP
$(\Theta ,\epsilon ,\delta )$-active PK DP [DMKB19]	Q,B	$✓$10	$✓$10	✗10	$(\Theta ,\epsilon ,\delta )$-APKDP ${\supset }^{\prec }$ $(\Theta ,\epsilon )$-NPr
$(\Theta ,\epsilon ,\delta )$-passive PK DP [DMKB19]	Q,B	$✓$10	$✓$10	✗10	$(\Theta ,\epsilon ,\delta )$-APKDP $\succ$ $(\Theta ,\epsilon ,\delta )$-PPKDP
					$(\Theta ,\epsilon )$-NPr ${\subset }^{\succ }$ $(\Theta ,\epsilon ,\delta )$-PPKDP
$(\Theta ,\Phi ,\epsilon )$-pufferfish Pr [KM12]	N,B	$✓$10	$✓$10	✗10	$(\Theta ,\epsilon )$-NPr $\subset$ $(\Theta ,\Phi ,\epsilon )$-PFPr $\supset$ $(ℛ,\epsilon )$-GcDP
$(\pi ,k,\epsilon )$-Bayesian DP [LL12]	N,B	$✓$10	$✓$10	✗10	BayDP[LL12] $\subset$ $(\Theta ,\Phi ,\epsilon )$-PFPr
$(\Theta ,\epsilon ,\delta )$-distribution Pr [JB14, KM19b]	Q,N,B	$✓$10	$✓$10	✗10	$(\Theta ,\epsilon ,\delta )$-DnPr $\supset$ $(\Theta ,\epsilon ,\delta )$-APKDP
$(\Theta ,\epsilon ,\delta )$-profile-based DP [GC19]	Q,N,B	$✓$10	$✓$10	✗10	$(\Theta ,\epsilon ,\delta )$-PBDP $\subset$ $(\Theta ,\epsilon ,\delta )$-DnPr
$(\Theta ,\epsilon ,\delta )$-probabilistic DnPr [KM19b]	Q,N,B	✗10	✗10	✗10	$\epsilon$-ProDP $\subset$ $(\Theta ,\epsilon ,\delta )$-PDnPr $\supset$ $(\Theta ,\epsilon )$-DnPr
$(f,\Theta ,\epsilon )$-divergence DnPr [KM19a]	Q,N,B	$✓$10	$✓$10	✗10	$(f,\epsilon )$-DP $\subset$ $(f,\Theta ,\epsilon )$-DDnPr $\supset$ $(\Theta ,\epsilon )$-DnPr
$(d,\Theta ,\epsilon )$-extended DnPr [KM19b]	N,V,B	$✓$10	$✓$10	✗10	$d_{𝒟}$-Pr $\subset$ $(d,\Theta ,\epsilon )$-EDnPr $\supset$ $(\Theta ,\epsilon )$-DnPr
$(d,f,\Theta ,\epsilon )$-ext. div. DnPr [KM19a]	Q,N,V,B	$✓$10	$✓$10	✗10	$(f,\Theta ,\epsilon )$-DDPr $\subset$ $(d,f,\Theta ,\epsilon )$-EDDnPr
					$(d,\Theta ,\epsilon )$-EDnPr $\subset$ $(d,f,\Theta ,\epsilon )$-EDDnPr
$\epsilon$-indistinguishable Pr [LXL13]	F	$✓$10	$✓$10	$✓$10	$\epsilon$-IndPr $\sim$ $\epsilon$-DP
$\epsilon$-semantic Pr [GKS08, KS14]	F	?	?	?	$\epsilon$-SemPr $\sim$ $\epsilon$-DP
$\epsilon$-range-bounded Pr [DR19]	F	?	?	?	$\epsilon$-RBPr $\sim$ $\epsilon$-DP
$f$-DP [DRS21]	Q,F	$✓$10	?	$✓$10	$f$-DP $\supset$ $(\epsilon ,\delta )$-DP
Gaussian DP [DRS21]	Q,F	$✓$10	?	$✓$10	GaussDP $\subset$ $f$-DP
weak-Federated $f$-DP [ZCLS21]	Q,F	$✓$10	?	$✓$10	$f$-wFDP $\subset$ GaussDP
strong-Federated $f$-DP [ZCLS21]	Q,F	$✓$10	?	$✓$10	$f$-wFDP $\prec$ $f$-sFDP $\subset$ GaussDP
$(\Theta ,\epsilon )$-aposteriori noiseless Pr [BBG+11]	B,F	$✓$10	$✓$10	?	$(\Theta ,\epsilon )$-ANPr $\sim$ $(\Theta ,\epsilon )$-NPr
$(\Theta ,\epsilon )$-positive membership Pr [LQS+13]	B,F	$✓$10	$✓$10	✗10	$(\Theta ,\epsilon )$-PMPr $\supset$ $\epsilon$-BoDP
$(\Theta ,\epsilon )$-negative membership Pr [LQS+13]	B,F	$✓$10	$✓$10	✗10	$(\Theta ,\epsilon )$-NMPr $\supset$ $\epsilon$-BoDP
$(\Theta ,\epsilon )$-membership Pr [LQS+13]	B,F	$✓$10	$✓$10	✗10	$(\Theta ,\epsilon )$-PMPr $\prec$ $(\Theta ,\epsilon )$-MPr $\succ$ $(\Theta ,\epsilon )$-NMPr
$(\Theta ,\epsilon ,\delta )$-adversarial Pr [RHMS09]	Q,B,F	$✓$10	$✓$10	✗10	$(\epsilon ,\delta )$-DP $\subset$ $(\Theta ,\epsilon ,\delta )$-AdvPr $\prec$ $(\Theta ,\epsilon )$-PMPr
$(\Theta ,\epsilon )$-information Pr [dPCF12]	N,B,F	?	?	?	$(\Theta ,\epsilon )$-InfPr $\succ$ $\epsilon$-DP
$(\text{Agg},\epsilon )$-zero-knowledge Pr [GLP11]	R	$✓$10	$✓$10	?10	$(\text{Agg},\epsilon )$-ZKPr $\succ$ $\epsilon$-DP
$({𝒫}_{\nu },\epsilon ,\delta )$-subspace DP [GGY21]	Q,R	?10	?	$✓$10	$({𝒫}_{\nu },\epsilon ,\delta )$-SubDP $\prec$ $(\epsilon ,\delta )$-DP
$(P,\epsilon )$-bounded leakage DP [LPR20]	Q,R	$✓$10	$✓$10	$✓$10	$(P,\epsilon )$-BLDP $\supset$ $(\epsilon ,\delta )$-DP
$(\Theta ,\Gamma ,\epsilon )$-coupled-worlds Pr [BGKS13]	N,B,R	$✓$10	$✓$10	✗10	$(\Theta ,\Gamma ,\epsilon )$-CWPr $\supset$ $\epsilon$-DP
$(\Theta ,\Gamma ,\epsilon ,\delta )$-inference CW Pr [BGKS13]	Q,N,B,F,R	?	?	✗10	$(\Theta ,\Gamma ,\epsilon ,\delta )$-IBCWPr $\succ$ $(\Theta ,\Gamma ,\epsilon )$-CWPr
$(\Theta ,\Gamma ,\epsilon ,\delta )$-inference DistDP [BGKS13]	Q,N,B,F,R	?	?	✗10	$(\Theta ,\epsilon )$-DDP $\succ$ $(\Theta ,\Gamma ,\epsilon ,\delta )$-IBDDP $\subset$ $(\Theta ,\Gamma ,\epsilon ,\delta )$-IBCWPr
Typical Stability [BF16]	Q,V,R	?	?	✗10	$(\Theta ,\gamma ,\epsilon ,\delta )$-TySt
${\epsilon }_{\kappa }$-SIM-computational DP [MPRV09]	C	$✓$10	$✓$10	$✓$10	${\epsilon }_{\kappa }$-SimCDP $\prec$ $\epsilon$-DP
${\epsilon }_{\kappa }$-IND-computational DP [MPRV09]	C	$✓$10	$✓$10	$✓$10	${\epsilon }_{\kappa }$-IndCDP $\prec$ ${\epsilon }_{\kappa }$-SimCDP
$(\text{Agg},\epsilon )$-computational ZK Pr [GLP11]	R,C	$✓$10	$✓$10	?	$(\text{Agg},\epsilon )$-CZKPr ${\supset }^{\prec }$ $(\text{Agg},\epsilon )$-ZKPr
$(\epsilon ,\delta ,f)$-output constrained DP [HMFS17]	N,V,C	$✓$10	$✓$10	$✓$10	$(\epsilon ,\delta ,f)$-OCDP $\supset$ ${\epsilon }_{\kappa }$-IndCDP
$(\epsilon ,\delta )$-DP for Record Linkage [HMFS17]	N,V,C	$✓$10	$✓$10	$✓$10	$(\epsilon ,\delta )$-RLDP $\subset$ $(\epsilon ,\delta ,f)$-OCDP

Table 3. Table 4: Queries for the literature review.

Query (BASE)	Hits
“differential privacy” relax	194
“differential privacy” variant -relax	225
Query (Google Scholar)	Hits
“differential privacy” “new notion”	332
“differential privacy” “new definition” -“new notion”	248

Equations

$$\mathbb{P}\left[A\in X\right]\le e^{\epsilon }\cdot \mathbb{P}\left[B\in X\right]\text{ and }\mathbb{P}\left[B\in X\right]\le e^{\epsilon }\cdot \mathbb{P}\left[A\in X\right].$$

$${\mathcal{L}}_{\mathcal{M}\left(D_1\right)/\mathcal{M}\left(D_2\right)}(O)=\ln \left(\frac{\mathbb{P}\left[\mathcal{M}\left(D_1\right)=O\right]}{\mathbb{P}\left[\mathcal{M}\left(D_2\right)=O\right]}\right).$$

$$\mathbb{P}\left[\mathcal{M}\left(D_1\right)\in S\right]\le e^{\epsilon }\cdot \mathbb{P}\left[\mathcal{M}\left(D_2\right)\in S\right]+\delta .$$

$$\mathbb{P}\left[\mathcal{M}\left(D_1\right)\in S\backslash S_1\right]\le e^{\epsilon }\cdot \mathbb{P}\left[\mathcal{M}\left(D_2\right)\in S\backslash S_1\right].$$

$${\mathbb{E}}_{O\sim \mathcal{M}\left(D_1\right)}\left[{\mathcal{L}}_{D_1/D_2}(O)\right]\le \epsilon .$$

$${\mathbb{E}}_{O\sim \mathcal{M}\left(D_1\right)}\left[e^{(\alpha -1){\mathcal{L}}_{D_1/D_2}(O)}\right]\le e^{(\alpha -1)\epsilon }.$$

$${\mathbb{E}}_{O\sim \mathcal{M}\left(D_1\right)}\left[e^{(\alpha -1){\mathcal{L}}_{D_1/D_2}(O)}\right]\le e^{(\alpha -1)(\xi +\rho \alpha )}.$$

$${\mathbb{E}}_{O\sim \mathcal{M}\left(D_1\right)}\left[f\left(e^{{\mathcal{L}}_{D_1/D_2}}\right)\right]\le \epsilon .$$

$$\mathbb{P}\left[\mathcal{M}\left(D_1\right)\in S\right]\le e^{\epsilon }\cdot \mathbb{P}\left[\mathcal{M}\left(D_2\right)\in S\right].$$

$$\mathbb{P}\left[\mathcal{M}\left(D_{-i}\right)\in S\right]\le e^{\Psi \left(i\right)}\mathbb{P}\left[\mathcal{M}\left(D\right)\in S\right]$$

$$\mathbb{P}\left[\mathcal{M}\left(D_1\right)\in S\right]\le e^{d_{\mathcal{D}}\left(D_1,D_2\right)}\cdot \mathbb{P}\left[\mathcal{M}\left(D_2\right)\in S\right].$$

$$M_{D_1,D_2}(\mathbb{P}\left[\mathcal{M}\left(D_1\right)\in S\right])\ge \mathbb{P}\left[\mathcal{M}\left(D_2\right)\in S\right]\text{and}{M}_{D_1,D_2}(\mathbb{P}\left[\mathcal{M}\left(D_1\right)\notin S\right])\ge \mathbb{P}\left[\mathcal{M}\left(D_2\right)\notin S\right]$$

$$\mathcal{M}{\left(D\right)}_{|D\sim \theta ,D(i)=t}{\approx }_{\epsilon }\mathcal{M}{\left(D\right)}_{|D\sim \theta ,D(i)=t^{\prime }}.$$

$$\mathcal{M}{\left(D\right)}_{|D\sim \theta ,D(i)=t}{\approx }_{\epsilon }\mathcal{M}{\left(D_{i\to t^{\prime }}\right)}_{|D\sim \theta ,D(i)=t}.$$

$${\mathcal{L}}_{i\leftarrow a/i\leftarrow b}^{\mathcal{M},\theta }(O,\hat{B})=\ln \left(\frac{{\mathbb{P}}_{(D,B)\sim \theta }\left[\mathcal{M}\left(D\right)=O|D(i)=a,B=\hat{B}\right]}{{\mathbb{P}}_{(D,B)\sim \theta }\left[\mathcal{M}\left(D\right)=O|D(i)=b,B=\hat{B}\right]}\right).$$

$${\mathbb{E}}_{(D,B)\sim {\theta }_{|D(i)=t,B=\hat{B}},O\sim \mathcal{M}\left(D\right)}\left[\max \left(0,1-e^{\epsilon -{\mathcal{L}}_{i\leftarrow t/i\leftarrow t^{\prime }}^{\mathcal{M},\theta }(O,\hat{B})}\right)\right]\le \delta .$$

$${\mathbb{E}}_{(D,B)\sim {\theta }_{|D(i)=t},O\sim \mathcal{M}\left(D\right)}\left[\max \left(0,1-e^{\epsilon -{\mathcal{L}}_{i\leftarrow t/i\leftarrow t^{\prime }}^{\mathcal{M},\theta }(O,B)}\right)\right]\le \delta .$$

$$\mathcal{M}{\left(D\right)}_{|D\sim \theta ,{\varphi }_1(D)}{\approx }_{\epsilon }\mathcal{M}{\left(D\right)}_{|D\sim \theta ,{\varphi }_2(D)}$$

$$\underset{S}{\inf }\left\{1-\mathbb{P}\left[\mathcal{M}\left(D_2\right)\in S\right]|\mathbb{P}\left[\mathcal{M}\left(D_1\right)\in S\right]\le x\right\}\ge f(x).$$

$$\frac{\mathbb{P}\left[t\in D|\mathcal{M}\left(D\right)=O\right]}{\mathbb{P}\left[t\notin D|\mathcal{M}\left(D\right)=O\right]}\le e^{\epsilon }\cdot \frac{\mathbb{P}\left[t\in D\right]}{\mathbb{P}\left[t\notin D\right]}\Rightarrow \mathbb{P}\left[t\in D|\mathcal{M}\left(D\right)=O\right]\le \frac{e^{\epsilon }\cdot \mathbb{P}\left[t\in D\right]}{1+\left(e^{\epsilon }-1\right)\mathbb{P}\left[t\in D\right]}.$$

$${\mathbb{P}}_{D\sim \theta }\left[t\in D|\mathcal{M}\left(D\right)\in S\right]$$

$${\mathbb{P}}_{D\sim \theta }\left[t\in D|\mathcal{M}\left(D\right)\in S\right]\le e^{\epsilon }\cdot {\mathbb{P}}_{D\sim \theta }\left[t\in D\right]+\delta .$$

$${D(i)}_{|D\sim \theta ,\mathcal{M}\left(D\right)\in S}{\approx }_{\epsilon }{D(i)}_{|D\sim \theta }.$$

$$\left|{\mathbb{P}}_{D\sim \theta }\left[D\in X|\mathcal{M}\left(D\right)\in S\right]-{\mathbb{P}}_{D\sim \theta }\left[D\in X|\mathcal{M}\left(D_{-i}\right)\in S\right]\right|\le \epsilon .$$

$${D(i)}_{|D\sim \theta ,\mathcal{M}\left(D\right)=O}{\approx }_{\epsilon }{D(i)}_{|D\sim \theta ,\mathcal{M}\left(D_{i\to b}\right)=O}.$$

$$\mathbb{P}\left[\mathcal{M}\left(D_1,r\right)\in S|P\left(D_1,r\right)=O_P\right]\le e^{\epsilon }\cdot \mathbb{P}\left[\mathcal{M}\left(D_2,r\right)\in S|P\left(D_2,r\right)=O_P\right]+\delta$$

$$\mathcal{M}{\left(D\right)}_{|D\sim \theta ,\text{priv}(D)=p}{\approx }_{\epsilon }\text{Sim}{\left(\text{agg}\left(D\right)\right)}_{|D\sim \theta ,\text{priv}(D)=p}$$

$${\text{priv}\left(D\right)}_{|(D,B)\sim \theta ,\mathcal{M}\left(D\right)=O,B=\hat{B}}{\approx }_{\left(\epsilon ,\delta \right)}{\text{priv}\left(D\right)}_{|(D,B)\sim \theta ,\text{Sim}\left(\text{agg}\left(D\right)\right)=O,B=\hat{B}}$$

$$\mathbb{P}\left[\Omega (\mathcal{M}\left(D_1\right))=1\right]\le e^{{\epsilon }_{\kappa }}\cdot \mathbb{P}\left[\Omega (\mathcal{M}\left(D_2\right))=1\right]+\text{neg}\left(\kappa \right)$$

$$\mathbb{P}\left[\Omega (\mathcal{M}\left(D\right))=1\right]-\mathbb{P}\left[\Omega ({\mathcal{M}}^{\prime }(D))=1\right]\le \text{neg}\left(\kappa \right)$$

Full text

These categories act like dimensions: variants from the same category cannot be combined, but variants from different categories can be combined to form new definitions. We also establish a partial ordering of relative strength between these notions by summarizing existing results. Furthermore, we list which of these definitions satisfy some desirable properties, like composition, post-processing, and convexity by either providing a novel proof or collecting existing ones.

1. 1 Introduction

2. 2 Differential Privacy

3. 2.1 The original version

4. 2.2 Dimensions

5. 2.3 Properties

6. 2.4 Relations between definitions

7. 3 Quantification of privacy loss (Q)

8. 3.1 Allowing a small probability of error

9. 3.2 Averaging the privacy loss

10. 3.3 Controlling the tail distribution of the privacy loss

11. 3.4 Extension

12. 3.5 Multidimensional definitions

13. 4 Neighborhood definition (N)

14. 4.1 Changing the sensitive property

15. 4.2 Limiting the scope of the definition

16. 4.3 Applying the definition to other types of input

17. 4.4 Extensions

18. 4.5 Multidimensional definitions

19. 5 Variation of privacy loss (V)

20. 5.1 Varying the privacy level across inputs

21. 5.2 Randomizing the variation of privacy levels

22. 5.3 Multidimensional definitions

23. 6 Background knowledge (B)

24. 6.1 Conditioning the output on the sensitive property

25. 6.2 Removing the effect of correlations in the data

26. 6.3 Multidimensional definitions

27. 7 Change in formalism (F)

28. 7.1 Hypothesis testing

29. 7.2 Changing the shape of the prior-posterior bounds

30. 7.3 Comparing two posteriors

31. 7.4 Multidimensional definitions

32. 8 Relativization of the knowledge gain (R)

33. 8.1 Taking into account auxiliary leakage function

34. 8.2 Borrowing concepts from zero-knowledge proofs

35. 8.3 Multidimensional definitions

36. 9 Computational power (C)

37. 9.1 Multidimensional definitions

38. 10 Summarizing table

39. 10.1 Proofs of properties

40. 10.1.1 Axioms

41. 10.1.2 Composition

42. 11 Scope and related work

43. 11.1 Methodology

44. 11.2 Out of scope definitions

45. 11.3 Local model and other contexts

46. 11.4 Related work

47. 12 Conclusion

1 Introduction

What does it mean for data to be anonymized? Samarati and Sweeney discovered that removing explicit identifiers from dataset records was not enough to prevent information from being re-identified [Sam01, Swe02], and they proposed the first definition of anonymization. This notion, called $k$-anonymity, is a property of a dataset: each combination of re-identifying fields must be present at least $k$ times. In the following decade, further research showed that sensitive information about individuals could still be leaked when releasing $k$-anonymous datasets, and many variants and definitions were proposed, such as $l$-diversity [MGKV06], $t$-closeness [LLV07], and $n$-confusion [ST12].

A common shortcoming of these approaches is that they defined anonymity as a property of the dataset: without knowing how the dataset is generated, arbitrary information can be leaked. This approach was changed with the introduction of differential privacy [DM05, Dwo06] (DP): rather than being a property of the sanitized dataset, anonymity was instead defined as a property of the process. It was inspired by Dalenius’ privacy goal that “Anything about an individual that can be learned from the dataset can also be learned without access to the dataset” [Dal77], a goal similar to one already used in probabilistic encryption [SM84].

Thanks to its useful properties, DP quickly became the flagship of data privacy definitions. Many algorithms and statistical processes were changed to satisfy DP and were adopted by organizations like Apple [Tea17], Facebook [MDH*+*20] [HDS*+*20], Google [ABC*+*20] [BDD*+*20] [BBD*+*21] [WZL*+*20], LinkedIn [RCM*+*20] [RSP*+*20], Microsoft [DKY17], OhmConnect [MP20], and the US Census Bureau [AAG*+*10] [GAP18] [FMM19].

Since the original introduction of differential privacy, many variants and extensions have been proposed to adapt it to different contexts or assumptions. These new definitions enable practitioners to get privacy guarantees, even in cases that the original DP definition does not cover well. This happens in a variety of scenarios: the noise mandated by DP can be too large and force the data custodian to consider a weaker alternative, the risk model might be inappropriate for certain use cases, or the context might require the data owner to make stronger statements on what information the privacy mechanism can reveal.

Figure 1 shows the prevalence of this phenomenon: approximately 225 different notions111We count all the definitions which are presented as “new” in the papers introducing them., inspired by DP, were defined in the last years. As we show in Figure 1, this phenomenon does not seem to slow down over time. These definitions can be extensions or variants of DP. An extension encompasses the original DP notion as a special case, while a variant changes some aspect, typically to weaken or strengthen the original definition.

With so many definitions, it is difficult for new practitioners to get an overview of this research area. Many definitions have similar goals, so it is also challenging to understand which are appropriate to use in which context. These difficulties also affect experts: a number of definitions listed in this work have been defined independently multiple times (often with identical meaning but different names, or identical names but different meanings). Finallly, variants are often introduced without a comparison to related notions.

This systematization of knowledge attempts to solve these problems. It is a taxonomy of variants and extensions of DP, providing short explanations of the intuition, use cases and basic properties of each. By categorizing these definitions, we attempt to simplify the understanding of existing variants and extensions, and of the relations between them. We hope to make it easier for new practitioners to understand whether their use case needs an alternative definition, and if so, which existing notions are the most appropriate, and what their basic properties are.

Contributions and organization

We systematize the scientific literature on variants and extensions of differential privacy, and propose a unified and comprehensive taxonomy of these definitions. We define seven dimensions: these are ways in which the original definition of DP can be modified or extended. We list variants and extensions that belong to each dimension, and we highlight representative definitions for each. Whenever possible, we compare these definitions and establish a partial ordering between the strengths of different notions. Furthermore, for each definition, we specify whether it satisfies Kifer et al.’s privacy axioms [KL10, KL12], (post-processing and convexity), and whether they are composable.

Our survey is organized as follows:

• •

In Section 2, we recall the original definition of DP and introduce our dimensions along which DP can be modified. Moreover, we present the basic properties of DP, and define how definitions can related to each other.

• •

In the following 7 sections (Sections 3 to 9), we introduce our dimensions, and list and compare the corresponding definitions.

• •

In Section 10, we summarize the results from the previous sections into a table, showing the corresponding properties with proofs, and list the known relations.

• •

In Section 11 we detail the methodology and scope of this work and review the related literature.

• •

Finally, in 12 we conclude this work.

2 Differential Privacy

In this chapter we recap the original DP definition with its basic properties, define how definitions can related to each other and introduce our dimensions along which DP can be modified.

Let $\mathcal{T}$ denote an arbitrary set of possible records. We typically use $t$ to denote the records themselves. A dataset is a finite indexed family of records. We denote by $\mathcal{D}$ space of possible datasets, individuals datasets are typically called $D$, $D^{\prime}$, $D_{1}$ or $D_{2}$. The indices of a dataset are typically called $i$ and $j$, with $D(i)$ referring to the $i$-th record of a dataset $D$. We denote by $D_{-i}$ the dataset $D$ whose $i$-th record has been removed.

Let $\mathcal{O}$ denote an arbitrary set of possible outputs; outputs are typically called $O$, and sets of outputs called $S$. A mechanism is a randomized function which takes a dataset as input and returns an output. Mechanisms are typically called $\mathcal{M}$ while $\mathcal{M}\left(D\right)$ is usually a random variable.

Probability distributions on $\mathcal{T}$ are called $\pi$, probability distribution on $\mathcal{D}$ are called $\theta$, and family of probability distributions on $\mathcal{D}$ are called $\Theta$. Given some property $\phi$, let $\mathcal{M}\left(D\right)_{|{}D\sim\theta,\phi}$ denote the random variable corresponding to the output of $\mathcal{M}\left(D\right)$, when $D$ is drawn from a distribution $\theta$ conditioned on $\phi$.

Table 1 summarizes the notations used throughout the paper.

2.1 The original version

The first DP mechanism, randomized response, was proposed in 1965 [War65], and data privacy definitions that are a property of a mechanism and not of the output dataset were already proposed in as early as 2003 [EGS03]. However, DP and the related notion of $\varepsilon$-indistinguishability were first formally defined in 2006 [Dwo06, DMNS06, DM05].

Definition 1** ($\varepsilon$-indistinguishability [DMNS06]).**

Two random variables $A$ and $B$ are $\varepsilon$-indistinguishable, denoted $A\approx_{\varepsilon}B$, if for all measurable sets $X$ of possible events:

[TABLE]

Informally, $A$ and $B$ are $\varepsilon$-indistinguishable if their distributions are “close”. This notion originates from the cryptographic notion of indistinguishability [GM84]. A similar notion, $(1,\varepsilon)$-privacy, was defined in [CM06], where $(1+\varepsilon)$ used in place of $e^{\varepsilon}$, and it was also called log-ratio distance in [HMSS19].

The notion of $\varepsilon$-indistinguishability is then used to define differential privacy.

Definition 2** ($\varepsilon$-differential privacy [Dwo06]).**

A privacy mechanism $\mathcal{M}$ is $\varepsilon$-differential private (or $\varepsilon$-DP) if for all datasets $D_{1}$ and $D_{2}$ that differ only in one record, $\mathcal{M}\left(D_{1}\right)\approx_{\varepsilon}\mathcal{M}\left(D_{2}\right)$.

Mechanisms

Besides the random response mechanism (which returns the true value with probability $p$ and returns a random value otherwise) in general there are three places where noise can be injected to guarantee DP [MTV*+*20]: it can be added to the input, to the output, and directly to the mechanism. For instance, in machine learning context input (e.g., [PAE*+*16]) and output (e.g., [PTB19]) pertubation are equivalent with sanitizing the dataset before and the predictions after the training respectively. Concerning mechanism pertubation, there are various techniques, such as loss function perturbation (e.g., [CMS11]), and gradient pertubation (e.g., [ACG*+*16]), which insert noise to the model objective and update respectively.

The most widely used distributions the noise is sampled from are Laplace, Gauss, and Exponential. Using the first and last makes any underlying mechanism to satisfy $\varepsilon$-DP for continious and discrete cases, however, they could result in large added noise. On the contrary, the middle distribution decrease the probability of such events significantly, but the obtained differential privacy guarantee is weaker (i.e. $\left(\varepsilon,\delta\right)$-DP, introduced in Section 3.

2.2 Dimensions

Variants and extensions of differential privacy modify the original definition in various ways. To establish a comprehensive taxonomy, a natural approach is to partition them into categories, depending on which aspect of the definition they change. Unfortunately, this approach fails for privacy definitions, many of which modify several aspects at once, so it is impossible to have a categorization such that every definition falls neatly into only one category.

The approach we take is to define dimensions along which the original definition can be modified. Each variant or extension of DP can be seen as a point in a multidimensional space, where each coordinate corresponds to one possible way of changing the definition along a particular dimension. To make this representation possible, our dimensions need to satisfy two properties:

• •

Mutual compatibility: definitions that vary along different dimensions can be combined to form a new, meaningful definition.

• •

Inner exclusivity: definitions in the same dimension cannot be combined to form a new, meaningful definition (but they can be pairwise comparable).

In addition, each dimension should be motivatable: there should be an intuitive explanation of what it means to modify DP along each dimension. Moreover, each possible choice within a dimension should be similarly understandable, to allow new practitioners to determine quickly which kind of definition they should use or study, depending on their use case.

We introduce our dimensions by reformulating the guarantee offered by DP, highlighting aspects that have been modified by its variants or extensions. Each dimension is attributed a letter, and we note the dimension letter corresponding to each highlight. This formulation considers the point of view of an attacker, trying to find out some sensitive information about some input data using the output of a mechanism.

An attacker with perfect background knowledge (B) and unbounded computation power (C) is unable (R) to distinguish (F) anything about an individual (N), uniformly across users (V) even in the worst-case scenario (Q).

This informal definition of DP with the seven highlighted aspects give us seven distinct dimensions. We denote each one by a letter and summarize them in Table 2. Each is introduced in its corresponding section.

Note that the interpretation of DP is subject to some debate. In [TSD20], authors summarize this debate, and show that DP can be interpreted under two possible lenses: it can be seen as an associative property, or as a causal property. The difference between the two interpretations is particularly clear when one supposes that the input dataset is modeled as being generated by a probability distribution.

• •

In the associative view, this probability distribution is conditioned upon the value of one record. If the distribution has correlations, this change can affect other records as well.

• •

In the causal view, the dataset is first generated, and the value of one record is then changed before computing the result of the mechanism.

While the causal view does not require any additional assumption to capture the intuition behind DP, the associative view requires that either all records are independent in the original probability distribution (the independence assumption), or the adversary must know all data points except one (the strong adversary assumption, which we picked in the reformulation above).

These considerations can have a significant impact on DP variants and extensions, either leading to distinct variants that attempt to capture the same intuition, or to the same variant being interpreted in different ways.

2.3 Properties

In this section, we introduce three main properties of differential privacy, that we then check against variants and extensions of DP listed in this work.

Privacy Axioms

Two important properties of data privacy notions are called privacy axioms, proposed in [KL10, KL12]. These are not axioms in a sense that they assumed to be true; rather, they are consistency checks: properties that, if not satisfied by a data privacy definition, indicate a flaw in the definition222The necessity of these were questioned in [HMQN13], where the authors showed a natural notions of anonymity that contradict them..

Definition 3** (Privacy axioms [KL10, KL12]).**

Post-processing333This definition must be slightly adapted for some variants, see for example Proposition 6 in Section 10.* (or transformation invariance): A privacy definition $\mathrm{Def}$ satisfies the post-processing axiom if, for any mechanism $\mathcal{M}$ satisfying $\mathrm{Def}$ and any probabilistic function $f$, the mechanism $D\rightarrow f(\mathcal{M}\left(D\right))$ also satisfies $\mathrm{Def}$.* 2. 2.

Convexity* (or privacy axiom of choice): A privacy definition $\mathrm{Def}$ satisfies the convexity axiom if, for any two mechanisms $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ satisfying $\mathrm{Def}$, the mechanism $\mathcal{M}$ defined by $\mathcal{M}\left(D\right)=\mathcal{M}_{1}(D)$ with probability $p$ and $\mathcal{M}\left(D\right)=\mathcal{M}_{2}(D)$ with probability $1-p$ also satisfies $\mathrm{Def}$.*

Most differential privacy variants and extensions, including the original definition of DP, satisfy these axioms, although some do not. We highlighted these in Table LABEL:tab:DP_variants in Section 10.

Composition

A third important property is one of differential privacy’s main strengths: composability. It guarantees that the output of two mechanisms satisfying a privacy definition still satisfies the definition, typically with a change in parameters. There are several types of composition: parallel composition, sequential composition, and adaptive composition. We introduce the first two below.

Theorem** (Parallel composition [Dwo06]).**

Let $\mathcal{M}_{1}$ be a $\varepsilon_{1}$-differentially private mechanism, and $\mathcal{M}_{2}$ a $\varepsilon_{2}$-differentially private mechanism. For any dataset $D$, let $D_{1}$ and $D_{2}$ be the result of an operation that separates records in two disjoint datasets. Then the mechanism $\mathcal{M}$ defined by $\mathcal{M}\left(D\right)=(\mathcal{M}_{1}(D_{1}),\mathcal{M}_{2}(D_{2}))$ is $\max(\varepsilon_{1},\varepsilon_{2})$-differentially private.

This property allows us to build locally differentially private mechanisms, in which a central server can compute global statistics without accessing the raw data from each user. In this work, we focus on sequential composition, which we simply call composition.

Theorem** (Sequential composition [Dwo06]).**

Let $\mathcal{M}_{1}$ be a $\varepsilon_{1}$-differentially private mechanism, and $\mathcal{M}_{2}$ a $\varepsilon_{2}$-differentially private mechanism. Then the mechanism $\mathcal{M}$ defined by $\mathcal{M}\left(D\right)=\left(\mathcal{M}_{1}(D),\mathcal{M}_{2}(D)\right)$ is $(\varepsilon_{1}+\varepsilon_{2})$-differentially private.

This theorem stays true if $\mathcal{M}_{2}$ depends on the value of $\mathcal{M}_{1}(D)$: this variant is called adaptative composition. This latter property allows to quantify the gain of information over time of an attacker interacting with a differentially private query engine.

In this work, we only consider sequential composition, in the more abstract form formalized below.

Definition 4** (Composability).**

A privacy definition $\mathrm{Def}$ with parameter $\alpha$ is composable if for any two mechanisms $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ satisfying respectively $\alpha_{1}$-$\mathrm{Def}$ and $\alpha_{2}$-$\mathrm{Def}$, the mechanism $\mathcal{M}\left(D\right)=\left(\mathcal{M}_{1}(D),\mathcal{M}_{2}(D)\right)$ satisfies $\alpha$-$\mathrm{Def}$ for some (non-trivial) $\alpha$.

2.4 Relations between definitions

When learning about a new data privacy notion, it is often useful to know what are the known relations between this notion and other definitions. However, definitions have parameters that often have different meanings, and whose value is not directly comparable. To capture extensions, when a definition can be seen as a special case of another, we introduce the following definition.

Definition 5** (Extensions).**

Let $\alpha\text{-}\mathrm{Def}_{1}$ and $\beta\text{-}\mathrm{Def}_{2}$ be data privacy definitions. We say that $\mathrm{Def}_{1}$ is extended by $\mathrm{Def}_{2}$, and denote is as $\mathrm{Def}_{1}\subset\mathrm{Def}_{2}$, if for all $\alpha$, there is a value of $\beta$ such that $\alpha$-$\mathrm{Def}_{1}$ is identical to $\beta$-$\mathrm{Def}_{2}$.

Concerning variants, to claim that a definition is stronger than another, we adopt the concept of ordering established in [CY16] using $\alpha$ and $\beta$ as tuples, encoding multiple parameters. Note that we slightly changed the original definition as that only required the second condition to hold, which would classify any extension as a stronger variant.

Definition 6** (Relative strength of privacy definitions).**

Let $\alpha\text{-}\mathrm{Def}_{1}$ and $\beta\text{-}\mathrm{Def}_{2}$ be data privacy definitions. We say that $\mathrm{Def}_{1}$ is stronger than $\mathrm{Def}_{2}$, and denote it $\mathrm{Def}_{1}\succ\mathrm{Def}_{2}$, if:

for all $\alpha$, there is a $\beta$ such that $\alpha\text{-}\mathrm{Def}_{1}\implies\beta\text{-}\mathrm{Def}_{2}$; 2. 2.

for all $\beta$, there is an $\alpha$ such that $\alpha\text{-}\mathrm{Def}_{1}\implies\beta\text{-}\mathrm{Def}_{2}$.

If $\mathrm{Def}_{1}$ is both stronger than and weaker than $\mathrm{Def}_{2}$, we say that the two definitions are equivalent, and denote it $\mathrm{Def}_{1}\sim\mathrm{Def}_{2}$.

Relative strength implies a partial ordering on the space of possible definitions. On the other hand, if two definitions are equivalent, this does not mean that they are equal: they could be only equal up to a change in parameters. Both relations are reflexive and transitive; and we define the symmetric counterpart of these relations as well (i.e., $\prec$ and $\supset$). Moreover, for brevity, we combine these two concepts in a single notation: if $\mathrm{Def}_{1}\subset\mathrm{Def}_{2}$ and $\mathrm{Def}_{1}\succ\mathrm{Def}_{2}$, we say that $\mathrm{Def}_{2}$ is a weaker extension of $\mathrm{Def}_{1}$, and denote it $\mathrm{Def}_{1}\subset^{\succ}\mathrm{Def}_{2}$.

A summarizing table is presented at the end of this work, where for each definition, we also highlight its dimensions and its relation to other notions. In Table LABEL:tab:DP_variants, we also specify whether these notions satisfy the privacy axioms and the composability property ($\checkmark$: yes, ✗: no, ?: currently unknown); in Section 10 we either provide a reference or a novel proof for each of these claims.

3 Quantification of privacy loss (Q)

The risk model associated to differential privacy is a worst-case property: it quantifies not only over all possible neighboring datasets but also over all possible outputs. However, in many real-life risk assessments, events with vanishingly small probability are ignored, or their risk weighted according to their probability. It is natural to consider analogous relaxations, especially since these relaxations often have better composition properties, and enable natural mechanisms like the Gaussian mechanism to be considered private [DR*+*14].

Most of the definitions within this section can be expressed using the privacy loss random variable, first defined in [DN03] as the adversary’s confidence gain, so we first introduce this concept. Roughly speaking, it measures how much information is revealed by the output of a mechanism.

Definition 7** (Privacy loss random variable [DN03]).**

Let $\mathcal{M}$ be a mechanism, and $D_{1}$ and $D_{2}$ two datasets. The privacy loss random variable between $\mathcal{M}\left(D_{1}\right)$ and $\mathcal{M}\left(D_{2}\right)$ is defined as:

[TABLE]

if neither $\mathbb{P}\left[\mathcal{M}\left(D_{1}\right)=O\right]$ nor $\mathbb{P}\left[\mathcal{M}\left(D_{2}\right)=O\right]$ is 0; in case only $\mathbb{P}\left[\mathcal{M}\left(D_{2}\right)=O\right]$ is zero then ${{\mathcal{L}}_{\mathcal{M}\left(D_{1}\right)/\mathcal{M}\left(D_{2}\right)}}(O)=\infty$, otherwise ${{\mathcal{L}}_{\mathcal{M}\left(D_{1}\right)/\mathcal{M}\left(D_{2}\right)}}(O)=-\infty$. When the mechanism is clear from context, we simply write ${{\mathcal{L}}_{D_{1}/D_{2}}}$.

Differential privacy bounds the maximum value of ${{\mathcal{L}}_{D_{1}/D_{2}}}$. Instead of considering the maximum value, which corresponds to the worst possible output, relaxations of this section will allow a small probability of error, consider the average of the privacy loss random variable, or describe its behavior in finer ways.

3.1 Allowing a small probability of error

The first option, whose introduction is commonly attributed to [DKM*+*06], relaxes the definition of $\varepsilon$-indistinguishability by allowing an additional small density of probability on which the upper $\varepsilon$ bound does not hold. This small density, denoted $\delta$, can be used to compensate for outputs for which the privacy loss is larger than $e^{\varepsilon}$. This led to the definition of approximate differential privacy, often simply called $\left(\varepsilon,\delta\right)$-DP. This is, by far, the most commonly used relaxation in the scientific literature. Several of the enlisted modifications from the other dimensions were introduced with this additive factor $\delta$, but for clarity we omit these details where not crutial.

Definition 8** ($\left(\varepsilon,\delta\right)$-differential privacy [DKM*+*06]).**

A privacy mechanism $\mathcal{M}$ is $\left(\varepsilon,\delta\right)$-DP if for any datasets $D_{1}$ and $D_{2}$ that differ only on one record, and for all $S\subseteq\mathcal{O}$:

[TABLE]

This definition is equivalent with Max-KL stability [BNS*+*16], a special case of algorithmic stability, which requires that one change in an algorithm’s inputs does not change its output “too much”.

The $\delta$ in $\left(\varepsilon,\delta\right)$-DP is sometimes explained as the probability that the privacy loss of the output is larger than $e^{\varepsilon}$ (or, equivalently, that the $\varepsilon$-indistinguishability formula is satisfied). In fact, this intuition corresponds to a different definition, first introduced in [MKA*+*08] as probabilistic DP (ProDP), also called $\left(\varepsilon,\delta\right)$-DP in distribution in [CO15]. A detailed explanation of the distinction between the two definitions can be found in [Mei18].

Definition 9** **($\left(\varepsilon,\delta\right)$-probabilistic differential

privacy [Mei18]).

A privacy mechanism $\mathcal{M}$ is $\left(\varepsilon,\delta\right)$-probabilistically DP (ProDP) if for any datasets $D_{1}$ and $D_{2}$ that differ only on one record there is a set $S_{1}\subseteq\mathcal{O}$ where $\mathbb{P}\left[\mathcal{M}\left(D_{1}\right)\in S_{1}\right]\leq\delta$, such that for all measurable sets $S\subseteq\mathcal{O}$:

[TABLE]

It is straightforward to show that $\left(\varepsilon,\delta\right)$-ProDP is stronger than $\left(\varepsilon,\delta\right)$-DP (with no change in parameters); a proof of the reverse result (with parameter change) is given in [ZWB*+*19]. Both definitions can be reformulated using the privacy loss random variable.

Theorem**.**

A mechanism $\mathcal{M}$ is:

• •

$\varepsilon$-DP $\Leftrightarrow$ $\mathbb{P}_{O\sim\mathcal{M}\left(D_{1}\right)}\left[{\mathcal{L}}_{D_{1}/D_{2}}(O)>\varepsilon\right]=0$ for all neighboring $D_{1}$ and $D_{2}$.

• •

$\left(\varepsilon,\delta\right)$-DP $\Leftrightarrow$ $\mathbb{E}_{O\sim\mathcal{M}\left(D_{1}\right)}\left[\max\left(0,1-e^{\varepsilon-{{\mathcal{L}}_{D_{1}/D_{2}}}(O)}\right)\right]\leq\delta$ for all neighboring $D_{1}$ and $D_{2}$.

• •

$\left(\varepsilon,\delta\right)$-ProDP $\Leftrightarrow$ $\mathbb{P}_{O\sim\mathcal{M}\left(D_{1}\right)}\left[{{\mathcal{L}}_{D_{1}/D_{2}}}(O)>\varepsilon\right]\leq\delta$ for all neighboring $D_{1}$ and $D_{2}$.

Approximate and probabilistic differential privacy can be combined to form $(\varepsilon,\delta_{a},\delta_{p})$-relaxed DP (RelDP) [ZQZ*+*15], which requires $(\varepsilon,\delta_{a})$-DP with probability at least $1-\delta_{p}$.

3.2 Averaging the privacy loss

As $\varepsilon$-DP corresponds to a worst-case risk model, it is natural to consider relaxations to allow for larger privacy loss for some outputs. It is also natural to consider average-case risk models: allowing larger privacy loss values only if lower values compensate it in other cases. One such relaxation is called Kullback-Leibler privacy [BD14, CY16]: it considers the arithmetic mean of the privacy loss random variable, which measures how much information is revealed when the output of a private algorithm is observed.

Definition 10** ($\varepsilon$-Kullback-Leibler privacy [BD14, CY16]).**

A privacy mechanism $\mathcal{M}$ is $\varepsilon$-Kullback-Leibler private (KLPr) if for all $D_{1}$, $D_{2}$ differing in one record:

[TABLE]

Note that this formula can be expressed as $D_{\text{KL}}\left(\mathcal{M}\left(D_{1}\right)|\mathcal{M}\left(D_{2}\right)\right)\leq\varepsilon$ where $D_{\text{KL}}$ is the Kullback-Leibler-divergence.

$\varepsilon$-KL privacy considers the arithmetic mean of the privacy loss random variable or, equivalently, the geometric mean of $e^{{{\mathcal{L}}_{D_{1}/D_{2}}}}$. This choice of averaging function does not attribute a lot of weight to worst-case events, where ${{\mathcal{L}}_{D_{1}/D_{2}}}$ takes high values. Rényi DP extends this idea by adding a parameter $\alpha\geq 1$, which allows controlling the choice of averaging function by bounding the $\alpha$th momentum of the privacy loss random variable.

Definition 11** ($(\alpha,\varepsilon)$-Rényi differential privacy [Mir17]).**

Given $\alpha>1$, a privacy mechanism $\mathcal{M}$ is $(\alpha,\varepsilon)$-Rényi DP (RenyiDP) if for all pairs of neighboring datasets $D_{1}$ and $D_{2}$:

[TABLE]

Note that this formula can be expressed as $D_{\alpha}\left(\mathcal{M}\left(D_{1}\right)|\mathcal{M}\left(D_{2}\right)\right)\leq\varepsilon$ where $D_{\alpha}$ is the Rényi-divergence of order $\alpha$.

This definition can be naturally extended by continuity to $\alpha=1$ (where it is equivalent to $\varepsilon$-KL privacy) and $\alpha=\infty$ (where it is equivalent to $\varepsilon$-DP). Larger values of $\alpha$ lead to more weight being assigned to worst-case events: $(\alpha,\varepsilon)$-Rényi DP $\succ$ $(\alpha^{\prime},\varepsilon)$-Rényi DP iff $\alpha>\alpha^{\prime}$. Besides $\alpha=1$ and $\alpha=\infty$, Rényi DP has a simple interpretation for some values of $\alpha$: $\alpha=2$ imposes a bound on the arithmetic mean of $e^{{{\mathcal{L}}_{D_{1}/D_{2}}}}$, $\alpha=3$ imposes it on the quadratic mean, $\alpha=4$ on the cubic mean, etc. A related technique is the moments accountant [ACG*+*16] which keeps track of a bound on the moments of the privacy loss random variable during composition.

It is possible to use other divergence functions to obtain other relaxations. For example, in [WBK19], the authors introduce two technical definitions, binary-$|\chi|^{\alpha}$ DP (b-$|\chi|^{\alpha}$ DP) and tenary-$|\chi|^{\alpha}$ DP (t-$|\chi|^{\alpha}$ DP), as part of a proof on amplification by sampling. Other examples of divergences can lead to other variants, like $\varepsilon$-total variation privacy [BD14] ($\varepsilon$-TVPr, using the total variance) and quantum DP [Col16] (QDP, using the quantum divergence).

3.3 Controlling the tail distribution of the privacy loss

Some definitions go further than simply considering a worst-case bound on the privacy loss, or averaging it across the distribution. They try to obtain the benefits of $\left(\varepsilon,\delta\right)$-DP with a smaller $\varepsilon$ which holds in most cases, but control the behavior of the bad cases better than $\left(\varepsilon,\delta\right)$-DP, which allows for catastrophic privacy loss in rare cases.

The first attempt to formalize this idea was proposed in [DR16], where the authors introduce concentrated DP (later renamed to mean-concentrated DP (mCoDP) in [BS16]). In this definition, a parameter controls the privacy loss variable globally, and another parameter allows for some outputs to have a greater privacy loss; while still requiring that the difference is smaller than a Gaussian distribution. In [BS16], the authors show that this definition does not satisfy the post-processing axiom, and propose another formalization of the same idea called zero-concentrated DP (zCoDP) [BS16], which requires that the privacy loss random variable is concentrated around zero.

Definition 12** ($(\xi,\rho)$-zero-concentrated differential privacy [BS16]).**

A mechanism $\mathcal{M}$ is $(\xi,\rho)$-zero-concentrated DP if for all pairs of neighboring datasets $D_{1}$ and $D_{2}$ and all $\alpha>1$:

[TABLE]

Four more variants of concentrated DP exist:

• •

$(\xi,\rho,\delta)$-approximate zero-concentrated DP [BS16] (AzCoDP), which relaxes $(\xi,\rho)$-zCoDP by only taking the Rényi divergence on events with probability higher than $1-\delta$ instead of on the full distribution.

• •

$(\xi,\rho,\omega)$-bounded CoDP [BS16] (bCoDP) relaxes $(\xi,\rho)$-zCoDP by requiring the inequality to hold only for $\alpha\leq\omega$.

• •

$(\rho,\omega)$-truncated CoDP [BDRS18] (tCoDP[BDRS18]) relaxes $(0,\rho)$-zCoDP in the same way.

• •

$(\xi,\tau)$-truncated CoDP [Col16] (tCoDP[Col16]) requires the Rényi divergence to be smaller than $\min(\xi,\alpha\tau)$ for all $\alpha\geq 1$.

The relations between these definitions and other notions in this section is well-understood. Besides the special cases (e.g., $(\rho,\infty)$-tCoDP[BDRS18] is the same as $(0,\rho)$-zCoDP) and the relations that are a direct consequence of the definitions (e.g., $(\xi,\rho)$-zCoDP is the same as the condition “$(\xi+\rho\alpha)$-RénDP for all $\alpha>0$”), we list known relations below.

Theorem**.**

For all $\varepsilon>0$, $\delta>0$, $\mu>0$, $\tau>0$, $\xi\geq 0$ and $\omega>1$:

• •

$\varepsilon$-DP $\Longrightarrow$ $\left(\frac{\varepsilon\left(e^{\varepsilon}-1\right)}{2},\varepsilon\right)$-mCoDP (Theorem 3.5 in **[DR16]**)

• •

$\varepsilon$-DP $\Longrightarrow$ $\left(0,\frac{\varepsilon^{2}}{2}\right)$-zCoDP (Lemma 8.3 in **[BS16]**)

• •

$\varepsilon$-DP $\Longleftrightarrow$ $(\varepsilon,0)$-zCoDP (Lemma 3.2 in **[BS16]**)

• •

$(\mu,\tau)$-mCoDP $\Longrightarrow$ $\left(\mu-\frac{\tau^{2}}{2},\frac{\tau^{2}}{2}\right)$-zCoDP (Lemma 4.2 in **[BS16]**)

• •

$(\xi,\rho)$-zCoDP $\Longrightarrow$ $\left(\xi+\rho,O(\sqrt{\xi+2\rho})\right)$-mCoDP (Lemma 4.3 in **[BS16]**)

• •

$(\xi,\rho)$-zCoDP $\Longrightarrow$ $\left(\xi+\rho+\sqrt{4\rho\log\left(\frac{\min\left(1,\sqrt{\pi\rho}\right)}{\delta}\right)},\delta\right)$-DP (Lemma 3.5 and 3.6 in **[BS16]**)

• •

$\left(\xi+\sqrt{\rho\log{\frac{1}{\delta}}}\right)$-DP $\Longrightarrow$ $\left(\xi-\frac{\rho}{4}+5\sqrt[4]{\rho},\frac{\rho}{4}\right)$-zCoDP (Lemma 3.7 in **[BS16]**)

• •

$(\rho,\omega)$-tCoDP**[BDRS18]** $\Rightarrow$ $\left(\hat{\varepsilon},\delta\right)$-DP, where $\hat{\varepsilon}=\rho+2\sqrt{\rho\log{\frac{1}{\rho}}}$ if $\log{\frac{1}{\delta}}\leq{(\omega-1)}^{2}\rho$, and $\hat{\varepsilon}=\rho\omega+\frac{\log{\frac{1}{\delta}}}{\omega-1}$ otherwise (Lemma 6 in **[BDRS18*]**) *

3.4 Extension

Most definitions of this section can be seen as bounding the divergence between $\mathcal{M}\left(D_{1}\right)$ and $\mathcal{M}\left(D_{2}\right)$, for different possible divergence functions. In [BD14], the authors use this fact to generalize them and define $(f,\varepsilon)$-divergence DP (DivDP), which takes the particular divergence used as a parameter $f$.

Definition 13** ($(f,\varepsilon)$-divergence differential privacy [BD14]).**

Let $f$ be a convex function such as $f(1)=0$. A privacy mechanism $\mathcal{M}$ is $(f,\varepsilon)$-divergence DP if for all pairs of neighboring datasets $D_{1}$, $D_{2}$:

[TABLE]

An instance of this definition was presented in [DR18] as $(f_{k},\varepsilon)$-divergence DP; which requires that $\mathbb{E}_{O\sim\mathcal{M}\left(D_{1}\right)}\left[{\left|e^{{\mathcal{L}}_{D_{1}/D_{2}}}-1\right|}^{k}\right]\leq\varepsilon^{k}$. This definition is mainly used to prove technical results on privacy/utility tradeoffs in the local model. For any $k\leq 1$, $\varepsilon$-DP implies $\left(f_{k},e^{\varepsilon}-1\right)$-DivDP, and when $k=2$, it is equivalent to $\left(2,\log\left(1+\varepsilon^{2}\right)\right)$-RényiDP (Section 2 in [DR18]).

Moreover, capacity bounded differential privacy (CBDP) was introduced in [CIM19], which uses $H$-restricted $f$-divergence: $D_{f}^{H}(P|Q)=\sup_{h\in H}[\mathbb{E}_{x\sim P}\left[h(x)\right]-\mathbb{E}_{x\sim Q}\left[f^{*}(h(x))\right]]$ where $f$ is a divergence, $H$ is a family of functions, and $f^{*}$ is the Fenchel conjugate444The Fenchel conjugate for a function $f$ with a domain $R$ is $f^{*}(x)=\sup_{y\in R}[xy-f(y)]$.. In other words, it requires the supremum condition to hold only for a selected set of functions (queries) instead of all possible ones.

Finally, most definitions in this section taking two real-valued parameters can be extended to use a family of parameters rather than a single pair of parameters. As shown in [SMM19] (Theorem 2) for approximate DP, probabilistic DP, and Rényi DP, finding the tightest possible family of parameters (for either definition) for a given mechanism is equivalent to specifying the behavior of its privacy loss random variable entirely.

3.5 Multidimensional definitions

Allowing a small probability of error $\delta$ by using the same concept as in $\left(\varepsilon,\delta\right)$-DP is very common; many new DP definitions were proposed in the literature with such a parameter. Unless it creates a particularly notable effect, we do not mention it explicitly and present the definitions without this parameter.

Definitions in this section can be used as standalone concepts: $\left(\varepsilon,\delta\right)$-DP is omnipresent in the literature, and the principle of averaging risk is natural enough for Rényi privacy to be used in practical settings, like posterior sampling [GSC17] or resistance to adversarial inputs in machine learning [PYGPA19]. Most variants in this section, however, are only used as technical tools to get better results on composition or privacy amplification [DR*+*14, WBK19, FMTT18, LK18].

4 Neighborhood definition (N)

The original definition of differential privacy considers datasets differing in one record. Thus, the datasets can differ in two possible ways: either they have the same size and differ only on one record, or one is a copy of the other with one extra record. These two options do not protect the same thing: the former protects the value of the records while the latter also protects their presence in the data: together, they protect any property about a single individual.

In many scenarios, it makes sense to protect a different property about their dataset, e.g., the value of a specific sensitive field, or entire groups of individuals. It is straightforward to adapt DP to protect different sensitive properties: all one has to do is change the definition of neighborhood in the original definition.

4.1 Changing the sensitive property

The original definition states that the $\varepsilon$-indistinguishability propery should hold for “any datasets $D_{1}$ and $D_{2}$ that differ only on the data of one individual”. Modifying the set of pairs $\left(D_{1},D_{2}\right)$ such that $\mathcal{M}\left(D_{1}\right)\approx_{\varepsilon}\mathcal{M}\left(D_{2}\right)$ is equivalent to changing the protected sensitive property.

Weaker relaxations

In DP, the difference between $D_{1}$ and $D_{2}$ is sometimes interpreted as “one record value is different”, or “one record has been added or removed”. In [KM11], the authors formalize these two options as bounded DP and unbounded DP. They also introduced attribute DP and bit DP, for smaller changes within the differing record.

Definition 14** ([KM11]).**

If a privacy mechanism $\mathcal{M}$ satisfies $\mathcal{M}\left(D_{1}\right)\approx_{\varepsilon}\mathcal{M}\left(D_{2}\right)$ for any pair $D_{1},D_{2}$, where $D_{1}$ can be obtained from $D_{2}$ by…

• •

…adding or removing one record, then $\mathcal{M}$ is $\varepsilon$-unbounded DP (uBoDP).

• •

…changing exactly one record, then $\mathcal{M}$ is $\varepsilon$-bounded DP (BoDP).

• •

…changing one attribute in a record, then $\mathcal{M}$ is $\varepsilon$-attribute DP (AttDP).

• •

…changing one bit of an attribute in a record, then $\mathcal{M}$ is $\varepsilon$-bit DP (BitDP).

In [KM11], authors show that $\varepsilon$-unbounded DP implies $2\varepsilon$-bounded DP, as changing a record can be seen as deleting it and adding a new one in its place. The original definition of $\varepsilon$-DP is the conjunction of $\varepsilon$-unbounded DP and $\varepsilon$-bounded DP. However, bounded DP is frequently used in the literature, especially when using local differential privacy, and often simply called differential privacy. It is also sometimes renamed, like in [FMTT18], where the authors call it per-person DP.

Another way to relax the neighborhood definition in DP is to consider that only certain types of information are sensitive. For example, if the attacker learns that their target has cancer, this is more problematic than if they learn that their target does not have cancer. This idea is captured in one-sided DP (OSDP) [KDH*+*20]: the neighbors of a dataset $D$ are obtained by replacing a single sensitive record with any other record (sensitive or not). The idea of sensitivity is formalized by a policy $P$, which specifies which records are sensitive. This idea cannot be captured simply by $\varepsilon$-indistinguishability, since one-sided DP is asymmetric.

Definition 15** ($(P,\varepsilon)$-one-sided differential privacy [KDH*+*20]).**

Given a policy $P\subseteq\mathcal{T}$, a privacy mechanism $\mathcal{M}$ is $(P,\varepsilon)$-one-sided DP iff for all datasets $D_{1}$ and $D_{2}$, where $D_{2}$ has been obtained by replacing a record $t\in D_{1}\cap P$ by any other record and for all $S\subseteq\mathcal{O}$:

[TABLE]

When $P=\mathcal{T}$, this is equivalent to bounded DP. Similar ideas were proposed in multiple papers:

• •

In [TCY21], the authors propose asymetric DP, which is the unbounded version of OSDP.

• •

In [APV19], the authors propose sensitive privacy, which determines which records are sensitive based on the data itself and a normality property $N$ and a graph-based definition of $k$-neighborhood, instead of using a data-independent determination.

• •

In [BSW18], the authors introduce anomaly-restricted DP, which assumes that there is only one outlier in the dataset, and that this outlier should not be protected.

Stronger notions

More restrictive definitions are also possible. First, some definitions make the definition of neighborhood more explicit when a single person can contribute multiple times to a dataset; this is the case for client/participant DP, defined in [MRTZ18]. Variants of differential privacy that do not protect individuals, but single contributions (in the case where the same person can contribute multiple times to a dataset), are also often used in practice, especially for machine learning applications [MRTZ18]. Some recent works also argue that in-between definitions are appropriate: rather than protecting a single contribution or entire users contributions, authors in [ADJ19] suggest that protecting elements that reveal information about users, after deduplicating or clustering contributions. For example, rather than protecting all website visits by a single user, or each visit individually, one might choose to protect the fact that a user ever visited a website (but not whether the user visited the same website once or many times). They call the corresponding definition element-level DP (ELDP).

In [Dwo08], the authors implicitly define $(c,\varepsilon)$-group privacy considers datasets that do not differ in one record, but possibly several, to protect multiple individuals. This can also be interpreted as taking correlations into account when using DP: DP under correlation [CFYD14] uses an extra parameter to describe the maximum number of records that the change of one individual can influence.

These two definitions are formally equivalent; but the implicit interpretation of DP behind them is different. $(c,\varepsilon)$-group privacy is compatible with the associative view under the strong adversary assumption (the adversary knows all records except $c$) or the causal view ($c$ records are changed after the data is generated). Meanwhile, DP under correlation implicitly considers the associative view with the independence assumption; and tries to relax that assumption. This last approach was further developed via dependent DP [LCM16], which uses “dependence relationships” to describe how much the variation in one record can influence the other records.

Definition 16** ($(R,c,\varepsilon)$-dependent differential privacy [LCM16]).**

A privacy mechanism $\mathcal{M}$ provides $(R,c,\varepsilon)$-dependent DP (DepDP) where $R$ is the probabilistic dependence relationship and $c$ is the dependence size, if for any pair of datasets $D_{1}$ and $D_{2}$, where $D_{2}$ has been obtained from $D_{1}$ by changing one record and the corresponding at most $c-1$ other records according to $R$, $\mathcal{M}\left(D_{1}\right)\approx_{\varepsilon}\mathcal{M}\left(D_{2}\right)$.

Note that when $R$ is the empty relation, or when $c=1$, this definition is equivalent to bounded DP: under the associative view of DP, this represents independence between records. Similar definitions appear in [WW21a, WW21b] as correlated indistinguishability and correlated tuple DP respectively, in which correlations are defined by a correlation matrix. Furthermore, in [WDN17, WWK*+*17] as correlated DP (CorDP), in which correlations are defined by an observation on other datasets, and in in [YSN15] as bayesian DP555There are two other notions with the same name: introduced in [TF20, LL12], we mention them in Section 5 and 6 respectively. (BayDP[YSN15]), where the neighborhood relation is defined by an adversary having some knowledge about correlations in the data. An extension is proposed in [LRYY19] as prior DP (PriDP) which considers a family of adversaries instead of a single adversary.

The strongest possible variant is considered in [KM11], where the authors define free lunch privacy (FLPr), in which the attacker must be unable to distinguish between any two datasets, even if they are completely different. This guarantee is a reformulation of Dalenius’ privacy goal [Dal77]; as such, all mechanisms that satisfy free lunch privacy have a near-total lack of utility.

Definition 17** ($\varepsilon$-free lunch privacy [KM11]).**

A privacy mechanism $\mathcal{M}$ satisfies $\varepsilon$-free lunch privacy if $\mathcal{M}\left(D_{1}\right)\approx_{\varepsilon}\mathcal{M}\left(D_{2}\right)$ for any pair of datasets $D_{1},D_{2}$.

4.2 Limiting the scope of the definition

Redefining the neighborhood property can also be used to reduce the scope of the definitions. In [SCDFSM17], the authors note that DP requires $\varepsilon$-indistinguishability of results between any pair of neighboring data sets, but in practice, the data custodian has only one data set $D$ they want to protect. Thus, they only require $\varepsilon$-indistinguishability between this data set $D$ and all its neighbors, calling the resulting definition individual DP (InDP). An equivalent definition was proposed in [CH16] as conditioned DP.

Definition 18** ($(D,\varepsilon)$-individual differential privacy [SCDFSM17]).**

Given a dataset $D\in\mathcal{D}$, a privacy mechanism $\mathcal{M}$ satisfies $(D,\varepsilon)$-individual DP if for any data set $D^{\prime}$ that differs in at most one record from $D$, $\mathcal{M}\left(D\right)\approx_{\varepsilon}\mathcal{M}\left(D^{\prime}\right)$.

This definition was further restricted in [RW21] where besides fixing a dataset $D$, a record $t$ is also fixed.

4.3 Applying the definition to other types of input

Many adaptations of DP are simply changing the neighborhood definition to protect different types of input data than datasets. A few examples follow.

• •

Location

In [EG16] the authors defined location privacy, in which neighbors are datasets which differ in at most one record, and the two differing reecords are at a physical distance smaller than a given threshold. This definition also appears in [CBY*+*18] as DP on $r$-location set666Distinct from DP on $\delta$-location set [XX15], which we mention in Chapter 5.. Several more location-related DP variants were defined in [NV20]: untrackability (which adopts differential privacy for set of locations by protecting whether they originated from a single user or by two users), undetectability and multi user untrackability, (which extend this idea further by not assuming both sets originated from the same private data and to multiple users respectively).

• •

Graph

In [HLMJ09] the authors adopt DP to graph- structured data and present multiple alternative definitions, which protect different parts of the graph: the strongest is node-DP, which protects a node and all its edges; the weakest is edge-DP, only protects one edge; and an intermediate definition is $k$-edge-DP, which protects a total of $k$ edges and nodes. Similarly to one-sided DP, in [TC12, KRWY16], the authors introduce out-link privacy, and protected DP which protects all outgoing edges from a given node and guarantees that no observer can learn much about the set of edges corresponding to any protected node respectively. In addition, in [Reu18], the author introduces $QL$-edged-labeled DP which only protecting a predetermined subset of outgoing edges. In [Pin18], the author introduces $l_{1}$-weighted DP, in which graph edges are weighted, and graphs are neighbors when the total weight of their differing edges is smaller than 1; this notion was also defined implicitly in [Sea16]. In [ZLBR20] the authors defined feasible-node DP for control-flow graphs where an addition or removal od a node results in a feasible run-time behaviors. In [SXK*+*19], the authors define decentralized DP which extends the graph neighborhood to two jumps. Finally, in [DWWG13] the authors introduce seamless privacy, which rather than protecting characteristics of a specific input graph, it ensures that certain pairs of queries on this graph return similar answers.

• •

Stream

Several authors adapt DP to a streaming context, where the attacker can access the mechanism’s internal states. In [DNPR10, DNP*+*10, Dwo10], authors define pan-privacy, which comes in two variants: event-level pan-privacy (called strong DP in [WSMD20]) protects individual events, and user-level pan-privacy protects all events associated to a single user. In [KPXP14], the authors extend the previous idea and propose $w$-event privacy, which protects any event sequence occurring within a window of at most $w$ timestamps. In [Far20, NV20] this was further extended to an infinite horizon via discounted DP (which keep assigning smaller-and-smaller weights to further-and-further events) and everlasting privacy (which limit the leakage of information users suffer, no matter how many executions a mechanism had), respectively. Finally, series-indistinguishability [WX17] captured data correlations in the streaming context and the authors in [LNP13, PHB21] adopted DP for Kalman-filters and to time-independent power spectral densities respectively.

• •

RAM and PIR

In [WCM18] the authors adopt DP for Random Access Memory and Private Information Retrieval. For RAM the neighborhood is defined over the sequence of logical memory requests over time; the same notion appears in [CCMS19] as differential obliviousness and in [ADK*+*19] as oblivious DP. The adaptation of neighborhood is similar in case of PIR; a similar notion appears in [TDG16] as $\varepsilon$-private PIR and in [PPY19] as $\varepsilon$-DPIR. Additionally, in [KKNO17], the authors use a similar idea to define DP for outsourced database systems.

• •

Text and Images

In [JLH19], the authors adapt DP for symbolic control systems, and introduce word-DP and substitution-word-DP, protecting respectively pairs of words whose Levenshtein distance is lower than a given parameter, or whose Hamming distance is lower than a given parameter. In [ZSZ*+*18], the authors adapt DP for text vectors, and propose text indistinguishability, in which the neighborhood relationship between two word vectors depends on their Euclidean distance. In [YWW13, Fan18, LDX*+*21], the authors define refinement DP, DP Image, and Pixel DP respectively, which adopts the definition for images with neighbors given by some transformation or metric.

• •

Miscellaneous

Beside the already mentioned fields DP was adopted to numerous other use-cases, such as for set operations in [YLL*+*17], for gossip protocols in [HD19], for functions in [Noz19], for genomic data in [SSB16], for recommendation systems in [GKPT15], for machine learning in [LBG17], and for bandit algorithms in [TD16, BDT19].

4.4 Extensions

It is natural to generalize the variants of this section to arbitrary neighboring relationships. One example is mentioned in [KM11], under the name generic DP777Another definition with the same name is introduced in [KL10, KL12], we mention it in Section 5. (GcDP[KM11]), where the neighboring relation is entirely captured by a relation $\mathcal{R}$ between datasets.

Definition 19** ($(\mathcal{R},\varepsilon)$-generic differential privacy [KM11]).**

Given a relation $\mathcal{R}\subseteq\mathcal{D}^{2}$, a privacy mechanism $\mathcal{M}$ satisfies $(\mathcal{R},\varepsilon)$-generic DP if for all $\left(D_{1},D_{2}\right)\in\mathcal{R}$, $\mathcal{M}\left(D_{1}\right)\approx_{\varepsilon}\mathcal{M}\left(D_{2}\right)$.

This definition is symmetric, but it can easily be modified to accommodate asymmetric definitions like one-sided DP.

Other definitions use different formalizations to also generalize the concept of changing the neighborhood relationship. Some (like pufferfish privacy, mentioned in Section 6) use pairs of predicates $\left(\phi_{1},\phi_{2}\right)$ that $D_{1}$ and $D_{2}$ must respectively satisfy to be neighbors. Others (like coupled-worlds privacy, mentioned in Section 8) use private functions, denoted priv, and define neighbors to be datasets $D_{1}$ and $D_{2}$ such as $\textsf{priv}\left(D_{1}\right)\neq\textsf{priv}\left(D_{2}\right)$ [BGKS13]. Others use a distance function $d$ between datasets, and neighbors are defined as datasets a distance lower than a given threshold $\Delta$; this is the case for DP under a neighborhood (DPUN), introduced in [FC14], adjacent DP (AdjDP), introduced in [KM20]888Originally simply called “differential privacy” by its authors., constrained DP (ConsDP), introduced in [ZLW09] (where the distance $d$ captures a utility-related constraint), and distributional privacy999Another definition with the same name is introduced in [Rot10, BLR13], we mention it in Section 5. (DlPr[ZLW09]), also introduced in [ZLW09] (with additional constraints on the neighborhood definition: neighboring datasets must be part of a fixed set $S_{\mathcal{D}}$ and have elements in common). This distance can also be defined as the sensitivity of the mechanism, like in sensitivity-induced DP [RA17] (SIDP), or implicitly defined by a set of constraints, like what is done implicitly in [KM11] via induced neighbors DP (INDP).

One notable instantiation of generic DP is blowfish privacy (BFPr) [HMD14]. Its major building blocks are a policy graph $G$, that specifies which pairs of domain values in $\mathcal{T}$ should not be distinguished between by an adversary; and a set of constraints $Q$ that specifies the set $\mathcal{I}_{Q}$ of possible datasets that the definition protects. It was inspired by the Pufferfish framework [KM12] (see Section 6), but the attacker is not assumed to have uncertainty over the data: instead, it models an attacker whose knowledge is a set of deterministic constraints on the data.

Definition 20** ($(G,\mathcal{I}_{Q},\varepsilon)$-blowfish privacy [HMD14]).**

Given a policy graph $G\in\mathcal{T}^{2}$ and a set of datasets $\mathcal{I}_{Q}$, a privacy mechanism $\mathcal{M}$ satisfies $(G,\varepsilon)$-blowfish privacy if for all datasets $D_{1}$ and $D_{2}$ in $\mathcal{I}_{Q}$ that differ in only one element $i$ such that $\left(D_{1}(i),D_{2}(i)\right)\in G$, $\mathcal{M}\left(D_{1}\right)\approx_{\varepsilon}\mathcal{M}\left(D_{2}\right)$.

As noted in [HMD14], a mechanism satisfies $\varepsilon$-bounded DP if and only if satisfies $(K,\mathcal{I}_{n},\varepsilon)$-blowfish privacy, where $K$ is the complete graph, and $\mathcal{I}_{n}$ is any datasets of size $n$. A particular instantiation of this idea is explored in [KM11] as induced DP (IndDP), where the definition of neighbors is induced by a set of constraints.

4.5 Multidimensional definitions

Modifying the protected property is orthogonal to modifying the risk model implied by the quantification of privacy loss: it is straightforward to combine these two dimensions. Indeed, many definitions mentioned in this section were actually introduced with a $\delta$ parameter allowing for a small probability of error. One-one particularly general and specific example is adjacency relation divergence DP [KM19a], and node-Rényi DP [DMS*+*21], which combines an arbitrary neighborhood definition (like in generic DP) with an arbitrary divergence function (like in divergence DP) and adopts Rényi DP to graphs respectively.

As the examples in Section 4.3 show, it is very common to change the definition of neighborhood in practical contexts to adapt what aspect of the data is protected. Further, local DP mechanisms like RAPPOR [EPK14] implicitly use bounded DP: the participation of one individual is not secret, only the value of their record is protected. Variants that limit the scope of the definition to one particular dataset or user, however, provide few formal guarantees and do not seem to be used in practice.

5 Variation of privacy loss (V)

In DP, the privacy parameter $\varepsilon$ is uniform: the level of protection is the same for all protected users or attributes, or equivalently, only the level of risk for the most at-risk user is considered. In practice, some users might require a higher level of protection than others or a data custodian might want to consider the level of risk across all users, rather than only considering the worst case. Some definitions take this into account by allowing the privacy loss to vary across inputs, either explicitly (by associating each user to an acceptable level of risk), or implicitly (by allowing some users to be at risk, or averaging the risk across users).

5.1 Varying the privacy level across inputs

In Section 4, we saw how changing the definition of the neighborhood can be used to adapt the definition of privacy and protect different aspects of the input data. However, the privacy protection in those variants is binary: either a given property is protected, or it was not. A possible option to generalize this idea further is to allow the privacy level to vary across possible inputs.

One natural example is to consider that some users might have higher privacy requirements than others, and make the $\varepsilon$ vary according to which user differs between the two datasets. This is done in personalized DP (PerDP), a notion first defined informally in [NAD14], then independently in [JYC15, ESS15, GR15, LWS15]. An equivalent notion is also defined in [AGK16] as heterogeneous DP, while a location-based definition is presented in [DA18] as personalized location DP. Note that in these definitions, the privacy level associated with each user is not considered sensitive, and cannot depend on the data itself.

Definition 21** ($\Psi$-personalized differential privacy [JYC15]).**

A privacy mechanism $\mathcal{M}$ provides $\Psi$-personalized DP if for every pair of neighboring datasets $\left(D,D_{-i}\right)$ and for all sets of outputs $S\subseteq\mathcal{O}$:

[TABLE]

where $\Psi$ is a privacy specification, which maps each index $i$ to the privacy level of the record $D(i)$.

As shown in [JYC15, CD20], $\varepsilon$-DP implies $\Psi$-PerDP, where $\Psi(i)=\varepsilon$ for all indices $i$; and $\Psi$-PerDP implies $\varepsilon$-DP where $\varepsilon=\max_{i}\Psi(i)$.

This definition can be seen as a refinement of the intuition behind one-sided DP, which separated records into sensitive and non-sensitive ones. The idea of making the privacy level vary across inputs can be generalized further, by also making the privacy level depend on the entire dataset and the data of the differing record. This is done in [LP15], where the authors define tailored DP (TaiDP).

Definition 22** ($\Psi$-tailored differential privacy [LP15]).**

A mechanism $\mathcal{M}$ satisfies $\Psi$-tailored differential privacy for $\Psi:\mathcal{T}\times\mathcal{D}\rightarrow\mathbb{R}_{0}^{\infty}$ if for any dataset $D$, $\mathcal{M}\left(D\right)\approx_{\varepsilon(D(i),D)}\mathcal{M}\left(D_{-i}\right)$.

A similar notion is input-discriminative DP [GLXC20] where $\Psi$ takes the two $\varepsilon$ values corresponding to the two dataset. The authors also defined minID-DP where $\Psi$ is the minimum function.

This concept can be applied to strengthen or weaken the privacy requirement for a record depending on whether they are an outlier in the dataset. In [LP15], the authors formalize this idea and introduce outlier privacy, which tailors an individual’s protection level to their “outlierness”. Other refinements are also introduced in [LP15]: simple outlier privacy (SOPr), simple outlier DP (SODP), and staircase outlier privacy (SCODP). A similar idea was explored in [KLL19], which introduced pareto DP (ParDP): it utilizes a pareto distribution of parameters $(p,r)$ to separate a large number of low-frequency individuals from a small number of high-frequency, and the sensitivity is calculated based on only the low-frequency individuals.

Finally, varying the privacy level across inputs also makes sense in continuous scenarios, where the neighborhood relationship between two datasets is not binary, but quantified. This is, for example, the case for $\varepsilon$-geo-indistinguishability [ABCP13], where two datasets $D_{1}$ and $D_{2}$ are considered $r$-neighbors if the only different record between $D_{1}$ and $D_{2}$ are at a distance $r$ of each other, and the $\varepsilon$ grows linearly with $r$.

5.2 Randomizing the variation of privacy levels

Varying the privacy level across inputs can also be done in a randomized way, by guaranteeing that some random fraction of users have a certain privacy level. One example is proposed in [HWR13] as random DP (RanDP): the authors note that rather than requiring DP to hold for any possible datasets, it is natural to only consider realistic datasets, and allow “edge-case” or very unrealistic datasets to not be protected. This is captured by generating the data randomly, and allowing a small proportion $\gamma$ of cases to not satisfy the $\varepsilon$-indistinguishability property.

Definition 23** ($(\pi,\gamma,\varepsilon)$-random differential privacy [HWR13]).**

Let $\pi$ be a probability distribution on $\mathcal{T}$, $D_{1}$ a dataset generated by drawing $n$ i.i.d. elements in $\pi$, and $D_{2}$ the same dataset as $D_{1}$, except one element was changed to a new element drawn from $\pi$. A mechanism $\mathcal{M}$ is $(\pi,\gamma,\varepsilon)$-random DP if $\mathcal{M}\left(D_{1}\right)\approx_{\varepsilon}\mathcal{M}\left(D_{2}\right)$, with probability at least $1-\gamma$ on the choice of $D_{1}$ and $D_{2}$.

The exact meaning of “with probability at least $1-\gamma$ on the choice of $D_{1}$ and $D_{2}$” can vary slightly. In [Hal12] and in [McC15], the authors introduce predictive DP (PredDP) and model-specific DP respectively, which quantify over all possible choices of $D_{1}$, and picks $D_{2}$ randomly in the neighborhood of $D_{1}$. In [DJRT13], $D_{1}$ and $D_{2}$ are both taken out of a set of density larger than $1-\gamma$, and the authors call this definition generalized DP (GdDP). The distribution generating the dataset is also not always assumed to be generating i.i.d. records; we denote the corresponding parameter by $\theta$.

Random DP might look similar to probabilistic DP: in both cases, there is a small probability that the privacy loss is unbounded. On the other hand, they are very different: in random DP, this probability is computed inputs of the mechanisms (i.e., users or datasets), for probabilistic DP, it is computed across mechanism outputs. Also similarly to probabilistic DP, excluding some cases altogether creates definitional issues: random DP does not satisfy the convexity axiom (see Proposition 4 in Section 10). We postulate that using a different tool to allow some inputs to not satisfy the mechanism, similar to approximate DP or Rényi DP, could solve this problem.

Usually, data-generating distributions are used for other purposes: they typically model an adversary with partial knowledge. However, definitions in this section still compare the outputs of the mechanisms given fixed neighboring datasets: the only randomness in the indistinguishability property comes from the mechanism. By contrast, definitions of Section 6 compare the output of the mechanism on a random dataset, so the randomness comes both from the data-generating distribution and the mechanism.

5.3 Multidimensional definitions

As varying the privacy level or limiting the considered datasets are two distinct way of relaxing differential privacy, it is possible to combine them with the previously mentioned dimensions.

Combination with N

The definitions described in Section 4 (e.g., generic DP or blowfish privacy) have the same privacy constraint for all neighboring datasets. Thus, they cannot capture definitions that vary the privacy level across inputs. On the other hand, they can be combined quite easily. For instance, [DA18] and [SPB21] adopted personalized DP to location and communication graphs respectively. Varying the privacy level across inputs also makes sense in “continuous” scenarios, where the neighborhood relationship between two datasets is not binary, but quantified. This is, for example, the case for geo-indistinguishability [ABCP13], where two datasets $D_{1}$ and $D_{2}$ are considered “$r$-neighbors” if the only different record between $D_{1}$ and $D_{2}$ are at a distance $r$ of each other, and the $\varepsilon$ grows linearly with $r$. Both ideas can be naturally captured together via distance functions. In [CABP13], the authors introduce $d_{\mathcal{D}}$-privacy ($d_{\mathcal{D}}$-Pr), in which the function $d_{\mathcal{D}}$ takes both datasets as input, and returns the corresponding maximal privacy loss (the $\varepsilon$) depending on the difference between the two datasets.

Definition 24** ($d_{\mathcal{D}}$-privacy [CABP13]).**

Let $d_{\mathcal{D}}:\mathcal{D}^{2}\rightarrow\mathbb{R}_{\infty}$. A privacy mechanism $\mathcal{M}$ satisfies $d_{\mathcal{D}}$-privacy if for all pairs of datasets $D_{1}$, $D_{2}$ and all sets of outputs $S\subseteq\mathcal{O}$:

[TABLE]

When $d_{\mathcal{D}}$ is proportional to the Hamiltonian difference between datasets, this is equivalent to $\varepsilon$-DP. In $d_{\mathcal{D}}$-privacy, the $d_{\mathcal{D}}$ function specifies both the privacy parameter and the definition of neighborhood: it can simply return $\infty$ on non-neighboring datasets, and vary the privacy level across inputs for neighboring datasets. In the original definition, the authors impose that $d_{\mathcal{D}}$ is symmetric, but this condition can also be relaxed to allow $d_{\mathcal{D}}$-privacy to extend definitions like one-sided DP.

Equivalent definitions of $d_{\mathcal{D}}$-privacy also appeared in [EG16] as $l$-privacy, and in [KM19b] as extended DP. Several other definitions, such as weighted DP [PGM14] (WeiDP), smooth DP [BD14] (SmoDP)101010Distinct from Smoothed DP [LX21] introduced later in this section. and earth mover’s privacy [FDM19] (EMDP), can be seen as particular instantiations of $d_{\mathcal{D}}$-privacy for specific functions $d$ measuring the distance between datasets. This is also the case for some definitions tailored for location privacy, like geo-graph-indistinguishability [TCAY19, TCAY20], which specifically applies to network graphs.

Random DP can also be combined with changing the neighborhood definition: in [XX15], the authors define DP on a $\delta$-location set111111Distinct from DP on $r$-location set [CBY*+*18], which we mention in Section 4., for which the neighborhood is defined by a set of “plausible” locations around the true location of a user. A notable definition using the same combination of dimensions is distributional privacy121212Another definition with the same name is introduced in  [ZLW09], we mention it in Section 4., introduced in [Rot10, BLR13]: it combines random DP (for a large family of distributions) and free lunch privacy.

Definition 25** ($(\varepsilon,\gamma)$-distributional privacy [Rot10, BLR13]).**

An algorithm $\mathcal{M}$ satisfies $(\varepsilon,\gamma)$-distributional privacy (DlPr[Rot10, BLR13]) if for any distribution $\pi$ over possible tuples, if $D_{1},D_{2}\in\mathcal{D}$ are picked by randomly drawing a fixed number $n$ of elements from $\pi$ without replacement, then with probability at least $1-\gamma$ over the choice of $D_{1}$ and $D_{2}$, $\mathcal{M}\left(D_{1}\right)\approx_{\varepsilon}\mathcal{M}\left(D_{2}\right)$.

Interestingly, this definition captures an intuition similar to the variants in 8: the adversary can only learn properties of the data-generating distribution, but not about particular samples (except with probability lower than $\gamma$). The authors also prove that if $\mathcal{M}$ is $(\varepsilon,\gamma)$-DlPr and $\gamma=o(\frac{1}{n^{2}})$, where $n$ is the size of the dataset being generated, then $\mathcal{M}$ is also $\varepsilon$-DP.

Combination with Q

Different risk models, like the definitions in Section 3, are also compatible with varying the privacy parameters across inputs. For example, in [Kre19], the author proposes endogeneous DP (EndDP), which is a combination of $\left(\varepsilon,\delta\right)$-DP and personalized DP. Similarly, pseudo-metric DP (PsDP), defined in [DNM*+*13], is a combination of $d_{\mathcal{D}}$-privacy and $\left(\varepsilon,\delta\right)$-DP; while extended divergence DP (EDivDP), defined in [KM19a], is a combination of $d_{\mathcal{D}}$-privacy and divergence DP.

Randomly limiting the scope of the definition can also be combined with ideas from the previous sections. For example, in [TF20], the authors introduce weak Bayesian DP (WBDP), which combines random DP and approximate DP. In [LX21] the authors introduced Smoothed DP131313Distinct from Smooth DP [BD14] introduced earlier in this chapter. which require $\left(\varepsilon,\delta\right)$-indistinguishability to hold for all the expected dataset from a set of distributions. In [WLF16], the authors introduce on average KL privacy, which uses KL-divergence as quantification metric, but only requires the property to hold for an “average dataset”, like random DP; a similar notion appears in [FS18] as average leave-one-out KL stability. In [TF20, DBB21], the authors introduce Bayesian DP141414There are two other notions with the same name: introduced in [YSN15, LL12], we mention them in Section 4 and 6 respectively. (BayDP[TF20]) and privacy at risk (PAR) respectively; both definitions combine random DP with probabilistic DP, with slightly different approaches: the former quantifies over all possible datasets and changes one fixed record randomly, while the latter selects both datasets randomly, conditioned on these datasets being neighbors.

In [KL10], Kifer et al. go further and generalize the intuition from generic DP, introduced in Section 4, and generalize the indistinguishability condition entirely. The resulting definition is also called generic differential privacy.

Definition 26** **($(\mathcal{R},M)$-generic differential

privacy [KL10, KL12]).

A privacy mechanism $\mathcal{M}$ satisfies $(\mathcal{R},M)$-generic DP (GcDP[KL10]) if for all measurable sets $S\subseteq\mathcal{O}$ and for all $\left(D_{1},D_{2}\right)\in\mathcal{R}$:

[TABLE]

where $M_{D_{1},D_{2}}:[0,1]\rightarrow[0,1]$ is a concave function continuous on $(0,1)$ such as $M_{D_{1},D_{2}}(1)=1$.

The privacy relation $\mathcal{R}$ is still the generalization of neighborhood and the privacy predicate is the generalization of the $\varepsilon$-indistinguishability to arbitrary functions. In particular, it can encompass all variants of described in Section 3 in addition to the ones in this section: for example, if $M_{D_{1},D_{2}}(x)=\min\left(1,xe^{\varepsilon}+\delta,1-(1-x-\delta)e^{-\varepsilon}\right)$ holds for for all $D_{1}$ and $D_{2}$, then this is equivalent to $\left(\varepsilon,\delta\right)$-DP. This definition was an attempt at finding the most generic definition that still satisfies privacy axioms: another extension defined in the same work, abstract DP (AbsDP) is even more generic, but no longer satisfies the privacy axioms.

Definitions in this section are particularly used in the context of local DP151515For details, see Section 11.3. and in particular for applications to location privacy: various metrics have been discussed to quantify how indistinguishable different places should be to provide users of a local DP mechanism with meaningful privacy protection [CEP*+*17].

6 Background knowledge (B)

In differential privacy, the attacker is implicitly assumed to have full knowledge of the dataset: their only uncertainty is whether the target belongs in the dataset or not. This implicit assumption is also present for the definitions of the previous dimensions: indeed, the attacker has to distinguish between two fixed datasets $D_{1}$ and $D_{2}$. The only source of randomness in $\varepsilon$-indistinguishability comes from the mechanism itself. In many cases, this assumption is unrealistic, and it is natural to consider weaker adversaries, who do not have full background knowledge. One of the main motivations to do so is to use significantly less noise in the mechanism [Dua09].

The typical way to represent this uncertainty formally is to assume that the input data comes from a certain probability distribution (named data evolution scenario in [KM12]): the randomness of this distribution models the attacker’s uncertainty. Informally, the more random this probability distribution is, the less knowledge the attacker has. However, the definition that follows depends whether DP is considered with the associative or the causal view. In the associative view, the sensitive property changes before the data is generated: it conditions the data-generating probability distribution. In the causal view, however, the sensitive property is only changed after the data is generated. The two options lead to very distinct definitions.

6.1 Conditioning the output on the sensitive property

Using a probability distribution to generate the input data means that the $\varepsilon$-indistinguishability property cannot be expressed between two fixed datasets. Instead, one natural way to express it is to condition this distribution on some sensitive property. The corresponding notion, noiseless privacy161616Another definition with the same name is introduced in [Far21], we mention it in Section 11.2. (NPr) was first introduced in [Dua09] and formalized in [BBG*+*11].

Definition 27** ($\left(\Theta,\varepsilon\right)$-noiseless privacy [Dua09, BBG*+*11]).**

Given a family $\Theta$ of probability distribution on $\mathcal{D}$, a mechanism $\mathcal{M}$ is $\left(\Theta,\varepsilon\right)$-noiseless private if for all $\theta\in\Theta$, all $i$ and all $t,t^{\prime}\in\mathcal{T}$:

[TABLE]

In the original definition, the auxiliary knowledge of the attacker is explicitly pointed out in an additional parameter. In the case where there is no $\delta$ of $(\varepsilon,0)$-DP, this syntactic add-on is not necessary [DMKB19], so we omitted it here.

This definition follows naturally from considering the associative view of DP with the strong adversary assumption, and attempting to relax this assumption. The exact way to model the adversary’s uncertainty can be changed; for example DP under sampling [LQS12], an instance of noiseless privacy, models it using random sampling.

This definition was not the first attempt at formalizing an adversary with restricted background knowledge. In [MGG09], authors define $\varepsilon$-privacy, which represents the background knowledge as a Dirichlet distribution (instead of an arbitrary distribution), whose parameters are interpreted as characteristics of the attacker. However, this definition imposes a condition on the output, but not on the mechanism which produced the output. As such, it does not offer strong semantic guarantees like the other definitions presented in this survey.

6.2 Removing the effect of correlations in the data

In [BGKS13], however, the authors argue that in the presence of correlations in the data, noiseless privacy can be too strong, and make it impossible to learn global properties of the data. Indeed, if one record can have an arbitrarily large influence on the rest of the data, conditioning on the value of this record can lead to very distinguishable outputs even if the mechanism only depends on global properties of the data. To fix this problem, they propose distributional DP (DistDP), an alternative definition that that only conditions the data-generating distribution on one possible value of the target record, and quantifies the information leakage from the mechanism.171717Note that the original formalization used in [BGKS13] was more abstract, and uses a simulator, similarly to variants introduced in Section 8. . In [DMKB19], the authors show that this creates definitional issues in the presence of background knowledge, and introduce causal DP (CausDP), to capture the same intuition without encountering the same problems.

Definition 28** ($\left(\Theta,\varepsilon\right)$-causal differential privacy [DMKB19]).**

Given a family $\Theta$ of probability distributions on $\mathcal{D}$, a mechanism $\mathcal{M}$ satisfies $\left(\Theta,\varepsilon\right)$-causal DP (CausDP) if for all probability distributions $\theta\in\Theta$, for all $i$ and all $t,t^{\prime}\in\mathcal{T}$:

[TABLE]

where $D_{i\rightarrow t^{\prime}}$ is the dataset obtained by changing the $i$-th record of $D$ into $t^{\prime}$.

In this definition, one record is changed after the dataset has been generated, so it does not affect other records through dependence relationships. These dependence relationships are the only difference between noiseless privacy and causal DP: when each record is independent of all others, this definition is equivalent to noiseless privacy [DMKB19].

6.3 Multidimensional definitions

Limiting the background knowledge of an attacker is orthogonal to the dimensions introduced previously: one can modify the risk model, introduce different neighborhood definitions, or even vary the privacy parameters across the protected properties along with limiting the attacker background knowledge.

Combination with Q

Modifying the risk model while limiting the attacker’s background knowledge has interesting consequences. In [DMKB19], the authors show that two options are possible: either consider the background knowledge as additional information given to the attacker or let the attacker influence the background knowledge. This distinction between an active and a passive attacker does not matter if only the worst-case scenario is considered, like in noiseless privacy. However, under different risk models, such as allowing a small probability of error, they lead to two different definitions.

Both of these definitions use an adapted version of the privacy loss random variable (PLRV) which explicitly models the attacker background knowledge: the data-generating distribution not only generates a dataset $D$ but also some auxiliary knowledge $B$, with values in a set $\mathcal{B}$.

Definition 29** (PLRV for partial knowledge [DMKB19]).**

Given a mechanism $\mathcal{M}$, a distribution $\theta$ with values in $\mathcal{D}\times\mathcal{B}$, an indice $i$, and values $a,b\in\mathcal{T}$, the PLRV of an output $O\in\mathcal{O}$ given partial knowledge $\hat{B}$ is:

[TABLE]

if the three conditions are satisfied:

$\mathbb{P}_{(D,B)\sim\theta}\left[D(i)=a,B=\hat{B}\right]\neq 0$* and $\mathbb{P}_{D\sim\theta}\left[D(i)=b,B=\hat{B}\right]\neq 0$* 2. 2.

$\mathbb{P}_{(D,B)\sim\theta}\left[\mathcal{M}\left(D\right)=O\;\middle|\;D(i)=a,B=\hat{B}\right]\neq 0$** 3. 3.

$\mathbb{P}_{(D,B)\sim\theta}\left[\mathcal{M}\left(D\right)=O\;\middle|\;D(i)=b,B=\hat{B}\right]\neq 0$.

If condition 1 does not hold, then ${\mathcal{L}}^{\mathcal{M},\theta}_{i\leftarrow a/i\leftarrow b}(O,B)=0$. Else, if condition 2 does not hold, then ${\mathcal{L}}^{\mathcal{M},\theta}_{i\leftarrow a/i\leftarrow b}(O,B)=-\infty$. Else, if condition 3 does not hold, then ${\mathcal{L}}^{\mathcal{M},\theta}_{i\leftarrow a/i\leftarrow b}(O,B)=\infty$.

This formalization can then be used to adapt noiseless privacy to a risk model similar to $\left(\varepsilon,\delta\right)$-DP, in the case of an active or a passive attacker. The active variant, active partial knowledge differential privacy (APKDP), quantifies over all possible values of the background knowledge. It was first introduced in [BBG*+*11, BGKS13] as noiseless privacy and reformulated in [DMKB19] to clarify that it implicitly assumes an active attacker.

Definition 30** **($\left(\Theta,\varepsilon,\delta\right)$-active partial knowledge differential

privacy [BBG*+*11, BGKS13, DMKB19]).

Given a family $\Theta$ of probability distribution on $\mathcal{D}\times\mathcal{B}$, a mechanism $\mathcal{M}$ is $\left(\Theta,\varepsilon,\delta\right)$-active partial knowledge DP (APKDP) if for all $\theta\in\Theta$, all indices $i$, all $t,t^{\prime}\in\mathcal{T}$, and all possible values $\hat{B}$ of the background knowledge:

[TABLE]

One specialization of this definition is DP under sampling [LQS12] (DPuS), which mandates DP to be satisfied after a random sampling is applied to the dataset. The authors use this definition to show that applying $k$-anonymity to a randomly sampled dataset provides differential privacy; but this definition could also be used on its own, to model the attacker’s uncertainty using a randomly sampled distribution.

The second definition, passive partial knowledge differential privacy [DMKB19] (PPKDP), is strictly weaker: it models a passive attacker, who cannot choose their background knowledge, and thus cannot influence the data. In this context, $\delta$ does not only apply to the output of the mechanism, but also to the value of the background knowledge.

Definition 31** ($\left(\Theta,\varepsilon,\delta\right)$-passive partial knowledge differential privacy [DMKB19]).**

Given a family $\Theta$ of probability distribution on $\mathcal{D}\times\mathcal{B}$, a mechanism $\mathcal{M}$ is $\left(\Theta,\varepsilon,\delta\right)$-passive partial knowledge DP (PPKDP) if for all $\theta\in\Theta$, all indices $i$, and all $t,t^{\prime}\in\mathcal{T}$:

[TABLE]

Causal DP can also be adapted to a risk model similar to $\left(\varepsilon,\delta\right)$-DP: in [BD19], authors introduce a similar notion to causal DP as inherit DP (InhDP), with the small difference that the second dataset is obtained by removing one record from the first dataset, instead of replacing it; and $\left(\varepsilon,\delta\right)$-indistinguishability is used. The authors also define empirical DP [BD19]181818Another definition with the same name is introduced in [ASV13], we mention it in Section 11.2., which is identical, except the empirical distribution is used instead of the actual data distribution, in context where the latter is unknown. In both cases, the influence of $\delta$ on the attacker model is unclear.

Combination with N

Modifying the neighborhood definition is simpler: it is clearly orthogonal to the dimensions introduced in this section. In all definitions of this section so far, the two possibilities between which the adversary must distinguish are similar to bounded DP. This can easily be changed to choose other properties to protect from the attacker. This is done in pufferfish privacy [KM12] (PFPr), which extends the concept of neighboring datasets to neighboring distributions of datasets.

Definition 32** ($(\Theta,\Phi,\varepsilon)$-pufferfish privacy [KM12]).**

Given a family of probability distributions $\Theta$ on $\mathcal{D}$, and a family of pairs of predicates $\Phi$ on datasets, a mechanism $\mathcal{M}$ verifies $(\Theta,\Phi,\varepsilon)$-pufferfish privacy if for all distributions $\theta\in\Theta$ and all pairs of predicates $\left(\phi_{1},\phi_{2}\right)\in\Phi$:

[TABLE]

Pufferfish privacy extends the concept of neighboring datasets to neighboring distributions of datasets; starting with a set of data-generating distributions, then conditioning them on sensitive attributes. The result compares pairs of distributions encompasses noiseless privacy, as well as other notions. For example, it captures bayesian DP191919There are two other notions with the same name: introduced in [YSN15, TF20], we mention them in Section 4 and 5 respectively. (BayDP[LL12]), introduced in [LL12], in which neighboring records have up to $k$ fixed elements in common and all other elements are generated randomly from a distribution $\pi$.

The same idea can be formalized by comparing pairs of distributions directly. This is done in [JB14, KM19b] via distribution privacy (DnPr). The two formalisms are equivalent: an arbitrary pair of distributions can be seen as a single distribution, conditioned on the value of a secret parameter. Distribution privacy was instantiated in [GC19] via profile-based DP (PBDP), in which the attacker tries to distinguish between different probabilistic user profiles. A similar idea was proposed in [BCDP19] as robust privacy, which uses lossy Wasserstein distance over the corresponding outputs to define the neighbourhood of the inputs.

Further relaxations encompassing the introduced dimensions are probabilistic distribution privacy [KM19b] (PDnPr), a combination of distribution privacy and probabilistic DP, extended distribution privacy [KM19b] (EDnPr), a combination of distribution privacy and $d_{\mathcal{D}}$-privacy, divergence distribution privacy [KM19a] (DDnPr), a combination of distribution privacy, and extended divergence distribution privacy [KM19a] (EDDnPr), which combines the latter two definitions. Finally, divergence distribution privacy with auxiliary inputs [KM19a] considers the setting where the attacker might not know the input probability distribution perfectly.

Definitions of this section are an active area of research; a typical question is to quantify in which conditions deterministic mechanisms can provide some level privacy. However, they are not used a lot in practice, likely because of their fragility: if the assumptions about the attacker’s limited background knowledge are wrong in practice, then the definitions do not provide any guarantee of protection.

7 Change in formalism (F)

The definition of differential privacy using $\varepsilon$-indistinguishability compares the distribution of outputs given two neighboring inputs. This is not the only way to capture the idea that a attacker should not be able to gain too much information on the dataset. Other formalisms have been proposed, which model the attacker more explicitly.

One such formalism reformulates DP in terms of hypothesis testing by limiting the type I and the type II error of the hypothesis that the output $O$ of a mechanism originates from $D_{1}$ (instead of $D_{2}$). Other formalisms model the attacker explicitly, by formalizing their prior belief as a probability distribution over all possible datasets. This can be done in two distinct ways. Some variants consider a specific prior (or family of possible priors) of the attacker, implicitly assuming a limited background knowledge, like in Section 6. We show that these variants can be interpreted as changing the prior-posterior bounds of the attacker. Finally, rather than comparing prior and posterior, a third formalism compares two possible posteriors, quantifying over all possible priors.

Definitions in this section provide a deeper understanding of the guarantees given by differential privacy, and some of them lead to tighter and simpler theorems on differential privacy, like composition or amplification results.

7.1 Hypothesis testing

First, differential privacy can be interpreted in terms of hypothesis testing [WZ10, KOV17]. In this context, an adversary who wants to know whether the output $O$ of a mechanism originates from $D_{1}$ (the null hypothesis) or $D_{2}$ (the alternative hypothesis). Calling $S$ the rejection region, the probability of false alarm (type I error), when the null hypothesis is true but rejected, is $\mathbb{P}_{FA}=\mathbb{P}\left[\mathcal{M}\left(D_{1}\right)\in S\right]$. The probability of missed detection (type II error), when the null hypothesis is false but retained, is $\mathbb{P}_{MD}=\mathbb{P}\left[\mathcal{M}\left(D_{2}\right)\in\mathcal{O}/S\right]$.

It is possible to use these probabilities, to reformulate DP:

[TABLE]

This hypotheses testing interpretation was used in [DRS21] to define $f$-differential privacy ($f$-DP), which avoids difficulties associated with divergence based relaxations. Specifically, its composition theorem is lossless as it provides a computationally tractable tool for analytically approximating the privacy loss. Moreover, there is a general duality between $f$-DP and infinite collections of $\left(\varepsilon,\delta\right)$-DP guarantees.

Definition 33** ($f$-differential privacy [DRS21]).**

Let $f:[0,1]\rightarrow[0,1]$ be a convex, continuous, and non-increasing function such that for all $x\in[0,1]$, $f(x)\leq 1-x$. A privacy mechanism $\mathcal{M}$ satisfies $f$-DP if for all neighboring $D_{1},D_{2}$ and all $x\in[0,1]$:

[TABLE]

Here, $S$ is the rejection region; and the infimum is the trade-off function between $\mathcal{M}\left(D_{1}\right)$ and $\mathcal{M}\left(D_{2}\right)$. The authors also introduce Gaussian differential privacy (GaussDP) as an instance of $f$-differential privacy, which tightly bounds from below the hardness of determining whether an individual’s data was used in a computation than telling apart two shifted Gaussian distributions. Besides, weak and strong federated f-DP [ZCLS21] are also defined that adopts the definition for federated learning [M*+*21]. They describe the privacy guarantee against an individual and a group of adversaries respectively.

7.2 Changing the shape of the prior-posterior bounds

Differential privacy can be interpreted as giving a bound on the posterior of a Bayesian attacker as a function of their prior. This is exactly the case in indistinguishable privacy (IndPr), an equivalent reformulation of differential privacy defined in [LXL13]: suppose that the attacker is trying to distinguish between two options $D=D_{1}$ and $D=D_{2}$, where $D_{1}$ corresponds to the option “$t\in D$” and $D_{2}$ to “$t\notin D$”. Initially, they associate a certain prior probability $\mathbb{P}\left[t\in D\right]$ to the first option. When they observe the output of the algorithm, this becomes the posterior probability $\mathbb{P}\left[t\in D|\mathcal{M}\left(D\right)=O\right]$. Combining the definition of $\varepsilon$-DP and the Bayes Theorem, we have202020Note that the original formalization used in [LXL13] was more abstract, and it used polynomially bounded adversaries what we introduce in Section 9.:

[TABLE]

A similar, symmetric lower bound can be obtained. Hence, differential privacy can be interpreted as bounding the posterior level of certainty of a Bayesian attacker as a function of its prior. We visualize these bounds on the first two figures of Figure 2.

Some variants of differential privacy use this idea in their formalism, rather than obtaining the posterior bound as a corollary to the classical DP definition. For example, positive membership privacy (PMPr) [LQS*+*13] requires that the posterior does not increase too much compared to the prior. Like noiseless privacy, it assumes an attacker with limited background knowledge.

Definition 34** ($\left(\Theta,\varepsilon\right)$-positive membership privacy [LQS*+*13]).**

A privacy mechanism $\mathcal{M}$ provides $\left(\Theta,\varepsilon\right)$-positive membership privacy if for any distribution $\theta\in\Theta$, any record $t\in\mathcal{D}$ and any $S\subseteq\mathcal{O}$:

[TABLE]

Note that this definition is asymmetric: the posterior is bounded from above, but not from below. In the same paper, the authors also define negative membership privacy (NMPr), which provides the symmetric lower bound, and membership privacy212121Another definition with the same name is introduced in [SDS*+*19], we mention it in Section 11.1. (MPr), which is the conjunction of positive and negative membership privacy. They show that this definition can represent differential privacy (in its bounded and unbounded variants), as well as other definitions like differential identifiability [LC12] and sampling DP [LQS12], which we mention in Section 11.2. Bounding the ratio between prior and posterior by $e^{\varepsilon}$ is also done in the context of location privacy: in [DGY*+*18], authors define $\varepsilon$-DP location obfuscation, which formalizes the same intuition as membership privacy.

A previous attempt at formalizing the same idea was presented in [RHMS09] as adversarial privacy (AdvPr). This definition is similar to positive membership privacy, except only the first relation is used, and there is a small additive $\delta$ as in approximate DP. We visualize the corresponding bounds on the third figure of Figure 2.

Definition 35** ($\left(\Theta,\varepsilon,\delta\right)$-adversarial privacy [RHMS09]).**

An algorithm $\mathcal{M}$ is $\left(\Theta,\varepsilon,\delta\right)$-adversarial private if for all $S\subseteq\mathcal{O}$, tuples $t$, and distributions $\theta\in\Theta$:

[TABLE]

Adversarial privacy (without $\delta$) was also redefined in [WXH17] as information privacy222222Another definition with the same name is introduced in [dPCF12], we mention it later in this section..

Finally, aposteriori noiseless privacy (ANPr) is a similar variant of noiseless privacy introduced in [BBG*+*11]; the corresponding bounds can be seen on the last figure of Figure 2.

Definition 36** ($\left(\Theta,\varepsilon\right)$-aposteriori noiseless privacy [BBG*+*11]).**

A mechanism $\mathcal{M}$ is said to be $\left(\Theta,\varepsilon\right)$-aposteriori noiseless private (ANPr) if for all $\theta\in\Theta$, all $S\subseteq\mathcal{O}$ and all $i$:

[TABLE]

We visualize the prior/posterior bounds for these various definitions in Figure 2.

7.3 Comparing two posteriors

In [GKS08, KS14], the authors propose an approach that captures an intuitive idea proposed by Dwork in [Dwo06]: “any conclusions drawn from the output of a private algorithm must be similar whether or not an individual’s data is present in the input or not”. They define semantic privacy (SemPr): instead of comparing the posterior with the prior belief like in DP, this bounds the difference between two posterior belief distributions, depending on which dataset was secretly chosen. The distance chosen to represent the idea that those two posterior belief distributions are close is the statistical distance. One important difference between the definitions in the previous subsection is that semantic privacy quantifies over all possible priors: like in DP, the attacker is assumed to have arbitrary background knowledge.

Definition 37** ($\varepsilon$-semantic privacy [GKS08, KS14]).**

A mechanism $\mathcal{M}$ is $\varepsilon$-semantically private if for any distribution over datasets $\theta$, any index $i$, any $S\subseteq\mathcal{O}$, and any set of datasets $X\subseteq\mathcal{D}$:

[TABLE]

A couple of other definitions also compare posteriors directly: inferential privacy [GK17] is a reformulation of noiseless privacy, and range-bounded privacy [DR19] (RBPr) requires that two different values of the PLRV are close to each other (instead of being between centered aroud zero like in $\varepsilon$-DP). It is equivalent to $\varepsilon$-DP up to a change in parameters, and is used as a technical tool to prove composition results.

7.4 Multidimensional definitions

This dimension could be combined with other dimensions fairly easily; several DP modifications from this section does belong to multiple dimensions, but to introduce the concept we overlooked these details.

Definitions that limit the background knowledge of the adversary explicitly formulate it as a probability distribution. As such, they are natural candidates for Bayesian reformulations. In [WXH17], the authors introduce identity DP, which is an equivalent Bayesian reformulation of noiseless privacy.

Another example is inference-based causal DP [DMKB19], similar to aposteriori noiseless DP, except it uses causal DP instead of noiseless DP.

Definition 38** ($\left(\Theta,\varepsilon\right)$-inference-based causal differential privacy [DMKB19]).**

Given a family $\Theta$ of probability distribution on $\mathcal{D}$, a mechanism $\mathcal{M}$ satisfies $\left(\Theta,\varepsilon\right)$-inference-based distributional DP if (IBCDP) for all probability distributions $\theta\in\Theta$, for all $i$, all $a,b\in\mathcal{T}$ and all outputs $O$:

[TABLE]

where $D_{i\rightarrow b}$ is the dataset obtained by changing the $i$-th record of $D$ into $b$.

The authors of semantic privacy also combined it with probabilistic DP, and simply called it $\left(\varepsilon,\delta\right)$-semantic privacy. Further, it is possible to consider different definitions of neighborhood. In [dPCF12], authors introduce information privacy232323 Another definition with the same name is introduced in [WXH17], we mention it earlier in this section. (InfPr), which can be seen as a posteriori noiseless privacy combined with free lunch privacy: rather than only considering the knowledge gain of the adversary on one particular user, it considers its knowledge gain about any possible group of values of the dataset.

Definition 39** ($\left(\Theta,\varepsilon\right)$-information privacy [dPCF12]).**

A mechanism $\mathcal{M}$ satisfies $\left(\Theta,\varepsilon\right)$-information privacy if for all probability distributions $\theta\in\Theta$, all $D\in\mathcal{D}$ and all $O\in\mathcal{O}$, $D_{|{}D\sim\theta}\approx_{\varepsilon}D_{|{}D\sim\theta,\mathcal{M}\left(D\right)=O}$.

The authors further prove that if $\Theta$ contains a distribution whose support is $\mathcal{D}$, then $\left(\Theta,\varepsilon\right)$-InfPr implies $2\varepsilon$-DP.

Apart from the hypothesis testing reformulations, that can be used to improve composition and amplification results, the definitions in this section mostly appear in theoretical research papers, to provide a deeper understanding of guarantees offered by DP and its alternatives. They do not seem to be used in practical applications.

8 Relativization of the knowledge gain (R)

A differentially private mechanism does not reveal more than a bounded amount of probabilistic information about a user. This view does not explicitly take into account other ways information can leak, like side-channel functions or knowledge about the structure of a social network. We found two approaches that attempt to include such auxiliary functions in DP variants. One possibility is to weaken DP by allowing or disregarding a certain amount of leakage; another option is to explicitly forbid the mechanism to reveal more than another function, considered to be safe for release.

8.1 Taking into account auxiliary leakage function

In [LPR20], authors define bounded leakage DP, which quantifies the privacy that is maintained by a mechanism despite bounded, additional leakage of information by some leakage function. Interestingly, this leakage function $P$ shares the randomness of the privacy mechanism: it can, for example, capture side-channel leakage from the mechanism’s execution. In the formal definition of this DP variant, the randomness is explicit: the privacy mechanism and the leakage takes the random bits $r\in{\{0,1\}}^{*}$ as an additional parameter.

Definition 40** ($\left(P,\varepsilon,\delta\right)$-bounded leakage differential privacy [LPR20]).**

Let $P:\mathcal{D}\times{\{0,1\}}^{*}$ be a leakage function. A privacy mechanism $\mathcal{M}$ is $\left(P,\varepsilon,\delta\right)$-bounded leakage differentially private (BLDP) if for all pairs of neighboring datasets $D_{1}$ and $D_{2}$, all outputs $O_{P}$ of $P$ such that $\mathbb{P}\left[P\left(D_{1},r\right)=O_{P}\right]\neq 0$ and $\mathbb{P}\left[P\left(D_{2},r\right)=O_{P}\right]\neq 0$, and all sets of outputs $S\subseteq\mathcal{O}$:

[TABLE]

where the randomness is taken over the random bits $r$.

As expected, if there is no leakage ($P$ is a constant function), this is simply $\left(\varepsilon,\delta\right)$-DP. The authors also show that it is closed for post-processing and composable. Furthermore, if the privacy mechanism is independent from the leakage function, it is strictly weaker than differential privacy.

8.2 Borrowing concepts from zero-knowledge proofs

When using the associative interpretation with the independence assumption, it is unclear how to adapt DP to correlated datasets like social networks: data about someone’s friends might reveal sensitive information about this person. The causal interpretation of DP does not suffer from this problem, but how to adapt the associative view to such correlated contexts? Changing the definition of the neighborhood is one possibility (see Section 4.1), but it requires knowing in advance the exact impact of someone on other records. A more robust option is to impose that the information released does not contain more information than the result of some predefined algorithms on the data, without the individual in question. The method for formalizing this intuition borrows ideas from zero-knowledge proofs [GMR89].

Instead of imposing that the result of the mechanism is roughly the same on neighboring datasets $D_{1}$ and $D_{2}$, the intuition is to impose that the result of the mechanism on $D_{1}$ can be simulated using only some information about $D_{2}$. The corresponding definition, called zero-knowledge privacy and introduced in [GLP11], captures the idea that the mechanism does not leak more information on a given target than a certain class of aggregate metrics. This class, called model of aggregate information in [GLP11], is formalized by a family of (possibly randomized) family of algorithms Agg.

Definition 41** ($\left(\textsf{Agg},\varepsilon\right)$-zero-knowledge privacy [GLP11]).**

Let Agg be a family of (possibly randomized) algorithms agg. A privacy mechanism $\mathcal{M}$ is $\left(\textsf{Agg},\varepsilon\right)$-zero-knowledge private (ZKPr) if there exists an algorithm $\textsf{agg}\in\textsf{Agg}$ and a simulator Sim such as for all datasets $D$ and indices $i$, $\mathcal{M}\left(D\right)\approx_{\varepsilon}\textsf{Sim}\left(\textsf{agg}\left(D_{-i}\right)\right)$.

In [GLP11], authors show that $\left(\textsf{Agg},\varepsilon\right)$-ZKPr implies $2\varepsilon$-DP for any Agg, while $\varepsilon$-DP implies $\left(\textsf{Agg},\varepsilon\right)$-ZKPr if the identity function is in Agg. This is yet another way to formalize the intuition that differential privacy protects against attackers who have full background knowledge.

Another approach is to ignore some leakage instead of limiting it. This is done in subspace DP [GGY21], which require DP to holds for a sub-space within the projection of the range of $\mathcal{M}$. Furthermore, induced subspace DP [GGY21] is also defined which ensures that the output of $\mathcal{M}$ does meet the invariant-linear external constraints. Essentially, they say for a mechanism to satisfy subspace DP, the function $D\rightarrow f(\mathcal{M}(D))$ must be DP for some function $f$.

8.3 Multidimensional definitions

Using a simulator allows making statements of the type “this mechanism does not leak more information on a given target than a certain class of aggregate metrics”. Similarly to noiseless privacy, it is possible to explicitly limit the attacker’s background knowledge using a data-generating probability distribution, as well as vary the neighborhood definitions to protect other types of information than the presence and characteristics of individuals. This is done in [BGKS13] as coupled-worlds privacy (CWPr), a generalization of distributional DP, where a family of functions priv represents the protected attribute.

Definition 42** ($(\Theta,\Gamma,\varepsilon)$-coupled-worlds privacy [BGKS13]).**

Let $\Gamma$ be a family of pairs of functions $(\textsf{agg},\textsf{priv})$. A mechanism $\mathcal{M}$ satisfies $(\Theta,\Gamma,\varepsilon)$-coupled-worlds privacy if there is a simulator Sim such that for all distributions $\theta\in\Theta$, all $(\textsf{agg},\textsf{priv})\in\Gamma$, and all possible values $p$:

[TABLE]

A special case of coupled-worlds privacy is also introduced in [BGKS13] as distributional DP, already mentioned in Section 6: each function priv captures the value of a single record, and the corresponding function agg outputs all other records.

Coupled-worlds privcay is a good example of combining variants from different dimensions: it changes several aspects of the original definition according to from N, B and R. Moreover, Q and F can easily be integrated with this definition by using $\left(\varepsilon,\delta\right)$-indistinguishability with a Bayesian reformulation. This is done explicitly in inference-based coupled-worlds privacy [BGKS13]; the same paper also introduces inference-based distributional differential privacy (IBDDP).

Definition 43** ($(\Theta,\Gamma,\varepsilon,\delta)$-inference-based coupled-worlds privacy [BGKS13]).**

Given a family $\Theta$ of probability distributions on $\mathcal{D}\times\mathcal{B}$, and a family $\Gamma$ of pairs of functions $(\textsf{agg},\textsf{priv})$, a mechanism $\mathcal{M}$ satisfies $(\Theta,\Gamma,\varepsilon,\delta)$-inference-based coupled-worlds privacy (IBCWPr) if there is a simulator Sim such that for all distributions $\theta\in\Theta$, and all $(\textsf{agg},\textsf{priv})\in\Gamma$:

[TABLE]

with probability at least $1-\delta$ over the choice of $O$ and $\hat{B}$.

Random DP was also combined with an idea similar to ZKPr: in [BF16], the authors introduce typical stability (TypSt), which combines random DP with approximate DP, except that rather using $\left(\varepsilon,\delta\right)$-indistinguishability between two outputs of the mechanism, it uses a simulator that only knows data-generating distribution.

Definition 44** ($(\Theta,\gamma,\varepsilon,\delta)$-typical stability [BF16]).**

Given a family $\Theta$ of probability distributions on $\mathcal{D}$, a mechanism $\mathcal{M}$ satisfies $(\Theta,\gamma,\varepsilon,\delta)$-typical stability (TypSt) if for all distributions $\theta\in\Theta$, there is a simulator Sim such that with probability at least $1-\gamma$ over the choice of $D\sim\theta$, $\mathcal{M}\left(D\right)\approx_{\varepsilon,\delta}\textsf{Sim}(\theta)$.

In the same paper, the authors introduce a variant of the same definition with the same name, which compares two outputs of the mechanism; this is essentially a combination between DlPr[Rot10, BLR13] and approximate DP.

We did not find any evidence that the variants and extensions of this section are used outside of theoretical papers exploring the guarantees they provide.

9 Computational power (C)

The $\varepsilon$-indistinguishability property in DP is information-theoretic: the attacker is implicitly assumed to have infinite computing power. This is unrealistic in practice, so it is natural to consider definitions where the attacker only has polynomial computing power. Changing this assumption leads to weaker data privacy definitions. In [MPRV09], two approaches have been proposed to formalize this idea: either modeling the distinguisher explicitly as a polynomial Turing machine, either allow a mechanism not to be technically differentially private, as long as one cannot distinguish it from a truly differentially private one.

Using a polynomial distinguisher on the output

The attacker is not explicit in the formalization of DP based on $\varepsilon$-indistinguishability. It is possible change the definition to make this attacker explicit: model it as a distinguisher, who tries to guess whether a given output $O$ comes from a dataset $D_{1}$ or its neighbor $D_{2}$. In doing so, it becomes straightforward to require this attacker to be computationally bounded: simply model it as a probabilistic polynomial-time Turing machine. In [MPRV09], the authors introduce IndCDP, short for Indistinguishability-based Computational DP.

Definition 45** ($\varepsilon_{\kappa}$-IndCDP [MPRV09]).**

A family ${\left(\mathcal{M}_{\kappa}\right)}_{\kappa\in\mathbb{N}}$ of privacy mechanisms $\mathcal{M}_{\kappa}$ provides $\varepsilon_{\kappa}$-IndCDP if there exists a negligible function neg such that for all non-uniform probabilistic polynomial-time Turing machines $\Omega$ (the distinguisher), all polynomials $p(\cdot)$, all sufficiently large $\kappa\in\mathbb{N}$, and all datasets $D_{1},D_{2}\in\mathcal{D}$ of size at most $p(\kappa)$ that differ only one one record:

[TABLE]

where neg is a function that converges to zero asymptotically faster than the reciprocal of any polynomial.

This definition can also be expressed using the authors define differential indistinguishability, a notion defined in [BKMR14] that adapts $\varepsilon$-indistinguishability to a polynomially bounded attacker.

Using a polynomial distinguisher on the mechanism

Another natural option is to require that the mechanism “looks like” a truly differentially private mechanism, at least to a computationally bounded distinguisher. In [MPRV09], the authors introduce SimCDP, short for Simulation-based Computational DP.

Definition 46** ($\varepsilon_{\kappa}$-SimCDP [MPRV09]).**

A family ${\left(\mathcal{M}_{\kappa}\right)}_{\kappa\in\mathbb{N}}$ of privacy mechanisms $\mathcal{M}_{\kappa}$ provides $\varepsilon_{\kappa}$-SimCDP if there exists a family ${\left(\mathcal{M}^{\prime}_{\kappa}\right)}_{\kappa\in\mathbb{N}}$ of $\varepsilon_{\kappa}$-DP and a negligible function neg such that for all non-uniform probabilistic polynomial-time Turing machines $\Omega$, all polynomials $p(\cdot)$, all sufficiently large $\kappa\in\mathbb{N}$, and all datasets $D\in\mathcal{D}$ of size at most $p(\kappa)$:

[TABLE]

where neg is a function that converges to zero asymptotically faster than the reciprocal of any polynomial.

In [MPRV09], the authors show that $\varepsilon_{\kappa}$-SimCDP implies $\varepsilon_{\kappa}$-IndCDP.

9.1 Multidimensional definitions

IndCDP has been adapted to different settings, and extended to arbitrary neighborhood relationships. In output constrained DP (OCDP), introduced in [HMFS17], the setting is a two-party computation, each party can have its own set of privacy parameters ($\varepsilon_{A}$, $\varepsilon_{B}$, $\delta_{A}$, and $\delta_{B}$—the $\delta$ parameters correspond to the $\textsf{neg}\left(\kappa\right)$ term in IndCDP), and the neighborhood relationship is determined by a function $f$. The authors also propose DP for Record Linkage (DPRL), an instance of OCDP that uses for a specific function $f$ that captures the need to protect non-matching records during the execution of a private record linkage protocol.

Some DP variants which explicitly model an adversary with a simulator can relatively easily be adapted to model a computationally bounded adversary, simply by imposing that the simulator must be polynomial. This is done explicitly in [GLP11], where the authors define computational zero-knowledge privacy (CZKPr), which could also be adapted to e.g., the two coupled-worlds privacy definitions as well.

Further, although we have not seen this done in practice in existing literature, the idea behind SimCDP can in principle be adapted to any other definition: rather than requiring that a given definition holds in an information-theoretic fashion, it should be possible to require that the mechanism “looks like” a mechanism which genuinely satisfies the definition.

Limiting the computational power of the attacker is a reasonable assumption, but for a large class of queries, it cannot provide significant benefits over classical DP in the typical client-server setting [GKY11]. Thus, existing work using it focuses on multi-party settings [BHE*+*18].

10 Summarizing table

In this section we summarize the results from the previous 7 sections into a table where we list the known relations and show the properties with either referring to the original proof or creating a novel one.

In Table LABEL:tab:DP_variants we list the differential privacy variants and extensions introduced in this work. For each, we specify their name, parameters and where they were introduced (column 1), which dimensions they belong to (column 2), which axioms they satisfy (column 3, post-processing on the left and convexity on the right), whether they are composable (column 4) and how they relate to other differential privacy notions (column 5). We do not list definitions whose only difference is that they apply DP to other types of input, like those from Section 4.3, or geolocation-specific definitions.

\endnote

See Proposition 1.\endnoteSee Proposition 2.\endnoteSee Proposition 3.\endnoteSee Proposition 4.\endnoteSee Proposition 5.\endnoteSee Proposition 6.\endnoteSee Proposition 7.\endnoteSee Proposition 8.\endnotePost-processing\endnoteConvexity\endnoteComposition\endnoteFollows from its

equivalence with $\varepsilon$-DP.\endnoteA proof appears in the

paper introducing the

definition.\endnoteA proof for a special case appears in the paper introducing the definition.\endnoteThis claim appears in [Mir17], its proof is in the unpublished full version.\endnoteAbbreviations used for dimensions:

• •

Q: Quantification of privacy loss

• •

N: Neighborhood definition

• •

V: Variation of privacy loss

• •

B: Background knowledge

• •

F: Formalism of privacy loss

• •

R: Relativization of knowledge gain

• •

C: Computational power

\setenotez

list-name=

\printendnotes

10.1 Proofs of properties

10.1.1 Axioms

Proposition 1**.**

All instantiations of DivDP satisfy both privacy axioms. In particular, approximate DP, MIDP, KLPr, RenDP, and zCoDP satisfy both axioms.

Proof.

The post-processing axiom follows directly from the monotonicity property of the $f$-divergence. The convexity axiom follows directly from the joint convexity property of the $f$-divergence. ∎

Proposition 2**.**

ProDP and ACoDP do not satisfy the convexity axiom.

Proof.

Consider the following mechanisms $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$, with input and output in $\{0,1\}$.

• •

$\mathcal{M}_{1}\left(0\right)=0$, $\mathcal{M}_{1}\left(1\right)=1$ with probability $\delta$, and $\mathcal{M}_{1}\left(1\right)=0$ with probability $1-\delta$.

• •

$\mathcal{M}_{2}\left(0\right)=\mathcal{M}_{2}\left(1\right)=1$.

Both mechanisms are $\left(\frac{1}{1-\delta},\delta\right)$-ProDP. Now, consider the mechanism $\mathcal{M}$ which applies $\mathcal{M}_{1}$ with probability $1-2\delta$ and $\mathcal{M}_{2}$ with probability $2\delta$. $\mathcal{M}$ is a convex combination of $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$, but the reader can verify that it is not $\left(\frac{1}{1-\delta},\delta\right)$-ProDP. The result for $(\xi,\rho,\delta)$-ACoDP is a direct corollary, since is is equivalent to $(\xi,\delta)$-ProDP when $\rho=0$. ∎

Proposition 3**.**

$d_{\mathcal{D}}$-Pr satisfies both privacy axioms. Further, EDivDP also satisfies both privacy axioms.

Proof.

The proof corresponding to PFPr in [KM12] is a proof by case analysis on every possible protected property. The fact that $\varepsilon$ is the same for every protected property has no influence on the proof, so we can directly adapt the proof to $d_{\mathcal{D}}$-Pr, and its combination with PFPr. Similarly, the proof can be extended to arbitrary divergence functions, like in Proposition 1. ∎

Proposition 4**.**

RanDP does not satisfy the convexity axiom.

Proof.

Let $\pi$ be the uniform distribution on $\{0,1\}$, let $D_{1}$ be generated by picking $10$ records according to $\pi$, and $D_{2}$ by flipping one record at random. Let $\mathcal{M}_{0}$ return [math] if all records are [math], and $\bot$ otherwise. Let $\mathcal{M}_{1}$ return $1$ if all records are $1$, and $\bot$ otherwise.

Note that both mechanisms are $(\pi,2^{-9},0)$-RanDP. Indeed, $\mathcal{M}_{0}$ will only return [math] for $D_{1}$ with probability $2^{-10}$, and for $D_{2}$ with probability $2^{-10}$ (if $D_{1}$ only has one $1$, which happens with probability $10\cdot 2^{-10}$, and this record is flipped, which happens with probability $0.1$). In both cases, $\mathcal{M}_{0}$ will return $\bot$ for the other database; which will be a distinguishing event. Otherwise, $\mathcal{M}_{0}$ will return $\tau$ for both databases, so $\mathcal{M}\left(D_{1}\right)\approx_{0}\mathcal{M}\left(D_{2}\right)$. The reasoning is the same for $\mathcal{M}_{1}$.

However, the mechanism $\mathcal{M}_{0.5}$ obtained by applying either $\mathcal{M}_{0}$ or $\mathcal{M}_{1}$ uniformly randomly doesn’t satisfy $(\pi,2^{-9},0)$-RanDP: the indinguishability property does not hold if $D_{1}$ or $D_{2}$ have all their records set to either [math] or $1$, which happens twice as often as either option alone. ∎

Proposition 5**.**

All variants of MPr, AdvPr, and ANPr satisfy both axioms. As a direct corollary, InfPr also satisfies both axioms.

Proof.

We prove it for AdvPr. A mechanism $\mathcal{M}$ satisfies $\left(\Theta,\varepsilon,\delta\right)$-AdvPr if for all $t\in\mathcal{T}$, $\theta\in\Theta$, and $S\subseteq\mathcal{O}$, $\mathbb{P}_{D\sim\theta}\left[t\in D\;\middle|\;\mathcal{M}\left(D\right)\in S\right]\leq e^{\varepsilon}\cdot\mathbb{P}_{D\sim\theta}\left[t\in D\right]+\delta$. We first prove that it satisfies the convexity axiom. Suppose $\mathcal{M}$ is a convex combination of $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$. Simplifying $\mathbb{P}_{D\sim\theta}\left[\dots\right]$ into $\mathbb{P}\left[\dots\right]$, we have:

[TABLE]

Denoting $X_{i}=\mathbb{P}\left[\mathcal{M}\left(D\right)\in S\text{ and }\mathcal{M}=\mathcal{M}_{i}\right]$ for $i\in\{1,2\}$, this gives:

[TABLE]

The proof for the post-processing axiom is similar, summing over all possible outputs $\mathcal{M}\left(D\right)$. It is straightforward to adapt the proof to all other definitions which change the shape of the prior-posterior bounds. ∎

Proposition 6**.**

Both versions of CDP satisfy both privacy axioms; where the post-processing axiom is modified to only allow post-processing with functions computable on a probabilistic polynomial time Turing machine. CZKPr also satisfies both privacy axioms.

Proof.

For Ind-CDP and the post-processing axiom, the proof is straightforward: if post-processing the output could break the $\varepsilon$-indistinguishability property, the attacker could do this on the original output and break the $\varepsilon$-indistinguishability property of the original definition.

For Ind-CDP and the convexity axiom, without loss of generality, we can assume that the sets of possible outputs of both mechanisms are disjoint (otherwise, this give strictly less information to the attacker). The proof is then the same as for the post-processing axiom.

For SimCDP, applying the same post-processing function to the “true” differentially private mechanism immediately leads to the result, since DP satisfies post-processing. The same reasoning holds for convexity.

The proof that CWPr satisfies both privacy axioms can be found in [BGKS13]; as an immediate corollary, CZKPr also satisfies both axioms. ∎

10.1.2 Composition

In this section, if $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ are two mechanisms, we denote $\mathcal{M}_{1+2}$ the mechanism defined by $\mathcal{M}_{1+2}(D)=\left(\mathcal{M}_{1}(D),\mathcal{M}_{2}(D)\right)$.

Proposition 7**.**

If $\mathcal{M}_{1}$ is $d^{1}_{\mathcal{D}}$-private and $\mathcal{M}_{2}$ is $d^{2}_{\mathcal{D}}$-private, then $\mathcal{M}_{1+2}$ is $d^{1+2}_{\mathcal{D}}$-private, where $d^{1+2}_{\mathcal{D}}\left(D_{1},D_{2}\right)=d^{1}_{\mathcal{D}}\left(D_{1},D_{2}\right)+d^{2}_{\mathcal{D}}\left(D_{1},D_{2}\right)$.

Proof.

The proof is essentially the same as for $\varepsilon$-DP. $\mathcal{M}_{1}$’s randomness is independent from $\mathcal{M}_{2}$’s, so:

[TABLE]

Most definition can also be combined with $d_{\mathcal{D}}$-privacy, and the composition proofs can be similarly adapted. ∎

Proposition 8**.**

In general, definitions which assume limited background knowledge from the adversary do not compose.

Proof.

The proof of Proposition 7 cannot be adapted to a context in which the attacker has limited background knowledge: as the randomness partially comes from the data-generating distribution, the two probabilities are no longer independent. A typical example considers two mechanisms which answer e.g., queries “how many records satisfy property $P$” and “how many records satisfy property $P$ and have an ID different from 4217”: the randomness in the data might make each query private, but the combination of two queries trivially reveals something about a particular user. Variants of this proof can easily be obtained for all definitions with limited background knowledge. ∎

11 Scope and related work

In this section, we detail our criteria for excluding particular data privacy definitions from our work, we list some relevant definitions that were excluded by this criteria, and we list related works and existing surveys in the field of data privacy.

11.1 Methodology

Whether a data privacy definition fits our description is not always obvious, so we use the following criterion: the attacker’s capabilities must be clearly defined, and the definition must prevent this attacker from learning about a protected property. Consequently, we do not consider:

• •

definitions which are a property of the output data and not of the mechanism;

• •

variants of technical notions that are not data privacy properties, like the different types of sensitivity;

• •

definitions whose only difference with DP is in the context and not in the formal property, like the distinction between local and global models.

In Section 11.2, we give a list of notions that we found during our survey, and considered to be out of scope for our book. To find a comprehensive list of DP notions, besides the definitions we were aware of or were suggested to us by experts, we conducted a wide literature review using two research datasets: BASE242424https://www.base-search.net/ and Google Scholar252525https://scholar.google.com/. The exact queries were run several times: November 1st in 2018, August 1st in 2019, June 1st in 2020, and 1st of September in 2021. The final result count are summarized in Table 4.

First, we manually reviewed each paper and filtered them out until we had only papers which either contained a new definition or were applying DP in a new setting. All papers which defined a variant or extension of DP are cited in this work.

11.2 Out of scope definitions

As detailed in the previous section, we considered certain data privacy definitions to be out of scope for our work, even when they seem to be related to differential privacy. This section elaborates on such definitions.

Lack of semantic guarantees

Some definitions do not provide clear semantic privacy guarantees, or are only used as a tool in order to prove links between existing definitions. As such, we did not include them in our survey.

• •

$\varepsilon$-privacy262626Another definition with the same name is introduced later in this section., introduced in [MGG09], was a first attempt at formalizing an adversary with restricted background knowledge by using Dirichlet distribution. This definition imposes a condition on the output, but not on the mechanism, consequently it does not offer strong semantic guarantees like noiseless privacy [Dua09, BBG*+*11] (introduced in Section 6).

• •

Relaxed indistinguishability, introduced in [RHMS09] is a relaxation of adversarial privacy that provides a plausible deniability by requiring for each tuple, that at least $l$ tuples must exist with $\varepsilon$-indistinguishability. However, it does not provide any guarantee against Bayesian adversaries.

• •

Mutual-information DP, introduced in [CY16] is an alternative way to average the privacy loss, similar to Section 3. It formalize the intuition that any individual record should not “give out too much information” on the output of the mechanism (or vice-versa).

• •

Individual privacy, introduce in [LGHC21] is also a mutual information based extension of DP which can encapsulate the (deterministic) secure multiparty computations [GM84], hence, we exclude it due to the lack of clear semantic privacy guarantees.

• •

Differential identifiability, introduced in [LC12], bounds the probability that a given individual’s information is included in the input datasets but does not measure the “change” in probabilities between the two alternatives. As such, it does not provide any guarantee against Bayesian adversaries272727Although it was reformulated in [LQS*+*13] as an instance of membership privacy introduced in Section 7..

• •

Crowd-blending privacy, introduced in [GHLP12], combines differential privacy with $k$-anonymity. As it is strictly weaker than any mechanism which always returns a $k$-anonymous dataset, the guarantees it provides against a Bayesian adversary are unclear. It is mainly used to show that combining crowd-blending privacy with pre-sampling implies zero-knowledge privacy [GHLP12, LP15].

• •

$(k,\varepsilon)$-anonymity, introduced in [HABMA17], first performs $k$-anonymisation on a subset of the quasi identifiers and then $\varepsilon$-DP on the remaining quasi-identifiers with different settings for each equivalence class of the $k$-anonymous dataset. The semantic guarantees of this definition are not made explicit.

• •

Integral privacy, introduced in [TNA16] looks at the inverse of the mechanism $\mathcal{M}$, and require all outputs $O$ to be generated by a large and diverse set of databases $D$. This could capture definitions like $k$-anonimity, but offer no semantic guarantee in the Bayesian sense.

• •

Membership privacy282828Another definition with the same name is introduced in [LQS*+*13], we mention it in Section 7, introduced in [SDS*+*19] is tailored to membership inference attacks on machine learning models; the guarantees it provides are not clear.

• •

Posteriori DP, introduced in [WYZ16], compares two posteriors in a way similar to inferential privacy in Section 7, but does not make the prior (and thus, the attacker model) explicit.

• •

$\varepsilon$-privacy292929Another definition with the same name is introduced earlier in this section., introduced in [PZJL20], adopts DP to location data and protect only the secret locations, but it only offers absolute bounds irrespectively of the prior.

• •

Noiseless privacy303030Another definition with the same name is introduced in [Dua09, BBG*+*11], we mention it in Section 6., and measure of privacy, introduced in [Far21] and [Far19] limits the change in the number of possible outputs when one record in the dataset changes and determines privacy using non-stochastic information theory respectively. Consequently, they do not bound the change in “probabilities” of the mechanism, so do not seem to offer clear guarantees against a Bayesian adversary.

• •

Weak DP, introduced in [WSMD20], adapts DP for streams, but it only provides a DP guarantee for the “average” of all possible mechanism outputs313131It also assumes that some uncertainty comes from the data itself, similarly to definitions in Section 7, rather than for the mechanism itself. Thus, its semantics guarantees are also unclear.

• •

Data-privacy and multi-dimensional data-privacy, introduce in [HCG18] and [SZH*+*20] ensures that an attacker’s inference accuracy and disclosure probability is below some thresholds.

• •

Error Preserving Privacy, introduced in [DNGRV18], states that the “variance” of the adversary’s error when trying to guess a given user’s record does not change significantly after. The exact adversary model is not specified.

Variants of sensitivity

A important technical tool used when designing differentially private mechanisms is the “sensitivity” of the function that we try to compute. There are many variants to the initial concept of global sensitivity [DMNS06], including local sensitivity [NRS07], smooth sensitivity [NRS07], restricted sensitivity [BBDS13], empirical sensitivity [CZ13], empirical differential privacy323232Even though it is introduced as a variant of DP, it was later shown to be a measure of sensitivity [CH16].333333Another definition with the same name is introduced in [BD19], we mention it in Section 6. [ASV13], recommen-dation-aware sensitivity [ZLR*+*13], record and correlated sensitivity [ZXLZ15], dependence sensitivity [LCM16], per-instance sensitivity [RW21], individual sensitivity [CD20], elastic sensitivity [JNS18], distortion sensitivity [BCDP19], and derivative & partial sensitivity [LPM18, MZU*+*21] respectively. We did not consider these notions as these do not modify the actual definition of differential privacy.

11.3 Local model and other contexts

In this book we focused on DP modifications typically used in the “global model”, in which a central entity has access to the whole dataset. It is also possible to use DP in other contexts, without formally changing the definition. The main alternative is the “local model”, where each individual randomizes their own data before sending it to an aggregator. This model is surveyed in [YLZ*+*20], and formally introduced in [DJW13].

Many definitions we listed were initially presented in the local model, such as $d_{\mathcal{D}}$-privacy [CABP13], geo-indistinguishability [ABCP13], earth mover’s Pr [FDM19], location Pr [EG16], profile-based DP [GC19], input-discriminative DP [GLXC20], divergence DP and smooth DP from [BD14], and extended DP, distribution Pr, and extended distribution Pr from [KM19b]. Below, we list the definitions that are the same as previously listed definitions, but used in a different attacker setting; the list also includes alternatives to the local and global models.

• •

In [SCR*+*11], the authors introduce distributed DP, which corresponds to local DP, with the additional assumption that only a portion of participants are honest.

• •

In [KPRU14], the authors define joint DP, to model a game in which each player cannot learn the data from any other player, but are still allowed to observe the influence of their data on the mechanism output.

• •

In [WHWX16], authors define a slightly different version of joined DP, called multiparty DP, in which the view of each “subgroup” of players is differentially private in respect to other players inputs.

• •

In [BEM*+*17], the authors define DP in the shuffled model, which falls in-between the global and the local model: the local model is augmented by an anonymous channel that randomly permutes a set of user-supplied messages, and differential privacy is only required of the output of the shuffler.

• •

In [BDT19], the authors defined local DP for bandits, a local version of instantaneous DP (mentioned in Section4).

• •

In [LKCT19], the authors define task-global DP and task-local DP, which are equivalents of element-level DP (mentioned in Section 4) in a meta-learning context.

• •

In [MK19], the authors define utility-optimized local DP, a local version of one-sided differential privacy (mentioned in Section 4) which additionally guarantees that if the data is considered sensitive, then a certain set of outputs is forbidden.

• •

In [DPZ*+*18, NYH*+*18, ABK*+*20, SXY21], the authors define personalized local DP, a local version of personalized DP (mentioned in Section 5).

• •

In [ACPP18], the authors define $d_{\mathcal{D}}$-local DP, a local version of $d_{\mathcal{D}}$-DP (mentioned in Section 5); this was defined as condensed local DP in [GTT*+*19].

• •

In [JLT18], the authors define localized information privacy, a local version of information privacy (mentioned in Section 7).

11.4 Related work

Some of the earliest surveys focusing on DP summarize algorithms achieving DP and applications [Dwo08, Dwo09]. The more detailed “privacy book” [DR*+*14] presents an in-depth discussion about the fundamentals of DP, techniques for achieving it, and applications to query-release mechanisms, distributed computations or data streams. Other recent surveys [NR19, RP21] focuses on the release of histograms and synthetic data with DP and describing existing privacy metrics and patterns while providing an overall view of different mathematical privacy preserving framework.

In [HZNF15], the authors classify different privacy enhancing technologies (PETs) into 7 complementary dimensions. Indistinguishability falls into the “Aim” dimension, but within this category, only $k$-anonymity and oblivious transfer are considered; differential privacy is not mentioned. In [AGM18], the authors survey privacy concerns, measurements and privacy-preserving techniques used in online social networks and recommender systems. They classify privacy into 5 categories; DP falls into “Privacy-preserving models” along with e.g., $k$-anonymity. In [WE18] the authors classified 80+ privacy metrics into 8 categories based on the output of the privacy mechanism. One of their classes is “Indistinguishability”, which contains DP as well as several variants. Some variants are classified into other categories; for example Rényi DP is classified into “Uncertainty” and mutual-information DP into Information gain/loss. The authors list 8 differential privacy variants; our taxonomy can be seen as an extension of the contents of their work (and in particular of the “Indistinguishability” category).

In [WYZ16], authors establish connections between differential privacy (seen as the additional disclosure of an individual’s information due to the release of the data), “identifiability” (seen as the posteriors of recovering the original data from the released data), and “mutual-information privacy” (which measures the average amount of information about the original dataset contained in the released data).

The relation between the main syntactic models of anonymity and DP was studied in [CT13], in which the authors claim that the former is designed for privacy-preserving data publishing (PPDP), while DP is more suitable for privacy preserving data mining (PPDM). The survey [ZZLZ20] investigate the issue of privacy loss due to data correlation under DP models and classify existing literature into three categories: using parameters to describe data correlation, using models to describe data correlation, and describing data correlation based on the framework.

Other surveys focus on location privacy. In [MH18], the authors highlight privacy concerns in this context and list mechanisms with formal provable privacy guarantees; they describe several variants of differential privacy for streaming (e.g., pan-privacy) and location data (e.g., geo-indistinguishability) along with extensions such as pufferfish and blowfish privacy. In [CEP*+*17], the authors analyze different kinds of privacy breaches and compare metrics that have been proposed to protect location data.

Finally, the appropriate selection of the privacy parameters for DP was also exhaustively studied. This problem in not trivial, and many factors can be considered: in [HGH*+*14], the authors used economic incentives, in [LC11, Kre19, PTB19], the authors looked at individual preferences, and in [LHC*+*19, LP19], the authors took into account an adversary’s capability in terms of hypothesis testing and guessing advantage respectively.

12 Conclusion

We proposed a classification of DP variants and extensions using the concept of dimensions. When possible, we compared definitions from the same dimension, and we showed that definitions from the different dimensions can be combined to form new, meaningful definitions. In theory, it means that even if there were only three possible ways to change a dimension (e.g., making it weaker or stronger), this would result in $3^{7}=2187$ possible definitions: the $\approx 225$ existing definitions shown in Figure 1 are only scratching the surface of the space of possible notions. Using these dimensions, we unified and simplified the different notions proposed in the literature. We highlighted their properties such as composability and whether they satisfy the privacy axioms by either collecting the existing results or creating new proofs, and whenever possible, we showed their relative relations to one another. We hope that this work will make the field of data privacy more organized and easier to navigate, especially for new practitioners.

Acknowledgments

The authors would like to thank James Bailie, David Basin, Alex Kulesza, Esfandiar Mohammadi, Jeremy Seeman, and the anonymous reviewers for their helpful comments. This work was partially funded by Google, and started when Balázs Pejó was at University of Luxembourg.