Local Distribution Obfuscation via Probability Coupling

TL;DR

This paper proposes a new local distribution obfuscation method using probability coupling that enhances privacy by adding noise proportional to distribution differences, optimizing utility while ensuring divergence-based privacy.

Contribution

It introduces a coupling mechanism for local distribution obfuscation that generalizes distribution privacy to divergence and optimizes utility using auxiliary information.

Findings

Probabilistic perturbation should be proportional to Earth mover's distance.

The coupling mechanism provides divergence distribution privacy.

The method balances privacy and utility effectively.

Abstract

We introduce a general model for the local obfuscation of probability distributions by probabilistic perturbation, e.g., by adding differentially private noise, and investigate its theoretical properties. Specifically, we relax a notion of distribution privacy (DistP) by generalizing it to divergence, and propose local obfuscation mechanisms that provide divergence distribution privacy. To provide f-divergence distribution privacy, we prove that probabilistic perturbation noise should be added proportionally to the Earth mover's distance between the probability distributions that we want to make indistinguishable. Furthermore, we introduce a local obfuscation mechanism, which we call a coupling mechanism, that provides divergence distribution privacy while optimizing the utility of obfuscated data by using exact/approximate auxiliary information on the input distributions we want to protect.

Tables

Table 1. TABLE II : Summary of basic properties of divergence DistP .

Sequential composition $\odot$ ($D_{\mathrm{KL}}$)	$A_b$ is $({\epsilon }_b,D_{\mathrm{KL}})$-DistP
	$\Rightarrow$ $A_1\odot A_0$ is $({\epsilon }_0+{\epsilon }_1,D_{\mathrm{KL}})$-DistP
Sequential composition $\bullet$ ($D_{\mathrm{KL}}$)	$A_b$ is $({\epsilon }_b,D_{\mathrm{KL}})$-DistP
	$\Rightarrow$ $A_1\bullet A_0$ is $({\epsilon }_0+{\epsilon }_1,D_{\mathrm{KL}})$-DistP
Post-processing	$A_0$ is $(\epsilon ,D_f)$-DistP $\Rightarrow$ $A_1\circ A_0$ is $(\epsilon ,D_f)$-DistP
Pre-processing (by $c$-stable $T$)	$A$ is $(\epsilon ,D)$-DistP $\Rightarrow$ $A\circ T$ is $(c\epsilon ,D)$-DistP

Table 2. TABLE III : Summary of basic properties of divergence XDistP .

Sequential composition $\odot$ ($D_{\mathrm{KL}}$)	$A_b$ is $({\epsilon }_b,W_{1,d},D_{\mathrm{KL}})$-XDistP
	$\Rightarrow$ $A_1\odot A_0$ is $({\epsilon }_0+{\epsilon }_1,W_{1,d},D_{\mathrm{KL}})$-XDistP
Sequential composition $\bullet$ ($D_{\mathrm{KL}}$)	$A_b$ is $({\epsilon }_b,W_{1,d},D_{\mathrm{KL}})$-XDistP
	$\Rightarrow$ $A_1\bullet A_0$ is $({\epsilon }_0+{\epsilon }_1,W_{1,d},D_{\mathrm{KL}})$-XDistP
Post-processing	$A_0$ is $(\epsilon ,W,D_f)$-XDistP ​$\Rightarrow$​ $A_1\circ A_0$ is $(\epsilon ,W,D_f)$-XDistP
Pre-processing (by $c$-stable $T$)	$A$ is $(\epsilon ,W,D)$-XDistP ​$\Rightarrow$​ $A\circ T$ is $(c\epsilon ,W,D)$-XDistP

Equations

$$\mathrm{Pr}[A(x)\in R]\le e^{\epsilon }\mathrm{Pr}[A(x^{\prime })\in R]+\delta$$

$$\mathrm{Pr}[A(x)\in R]\le e^{\epsilon d(x,x^{\prime })}\mathrm{Pr}[A(x^{\prime })\in R]+\delta$$

$$D_{\infty }^{\delta }(\mu \parallel {\mu }^{\prime })=\underset{\begin{array}{c}R\subseteq \mathsf{supp}(\mu ),\mu [R]\ge \delta \end{array}}{\max }\ln \frac{\mu [R]-\delta }{{\mu }^{\prime }[R]}.$$

$$\mathcal{F}=\{f:{\mathbb{R}}^{+}\to {\mathbb{R}}^{+}\mid f\text{mbox}isconvexandf(1)=0\}.$$

$$~{}~{}~{}~{}\mathit{D}_{f}(\mu\parallel\mu^{\prime})=\sum_{\begin{subarray}{c}y\in\mathsf{supp}(\mu^{\prime})\end{subarray}}\mu^{\prime}[y]\,f\bigl{(}\textstyle\frac{\mu[y]}{\mu^{\prime}[y]}\bigr{)}{.}$$

$$\mathit{W}_{p,\mathit{d}}(\lambda,\mu)=\min_{\gamma\in\mathsf{cp}(\lambda,\mu)}\bigl{(}\sum_{\hskip 18.08331pt(x_{0},x_{1})\in\mathsf{supp}(\gamma)}\mathit{d}(x_{0},x_{1})^{p}\gamma[x_{0},x_{1}]\bigr{)}^{\!\frac{1}{p}}{.}$$

$$W_{1,d}(\lambda ,\mu )=\underset{\gamma \in \mathsf{cp}(\lambda ,\mu )}{\min }\sum _{(x_0,x_1)\in \mathsf{supp}(\gamma )}d(x_0,x_1)\gamma [x_0,x_1].$$

$$\mathit{\Gamma_{\!p,\mathit{d}}}(\lambda,\mu)=\operatornamewithlimits{argmin}_{\gamma\in\mathsf{cp}(\lambda,\mu)}\bigl{(}\sum_{\hskip 18.08331pt(x_{0},x_{1})\in\mathsf{supp}(\gamma)}\mathit{d}(x_{0},x_{1})^{p}\gamma[x_{0},x_{1}]\bigr{)}^{\!\frac{1}{p}}{.}$$

$$D(A^{\#}(\lambda )\parallel A^{\#}({\lambda }^{\prime }))\le \epsilon .$$

$$D(A^{\#}(\lambda )\parallel A^{\#}({\lambda }^{\prime }))\le \epsilon d(\lambda ,{\lambda }^{\prime }).$$

$$D_f(A^{\#}({\lambda }_0)\parallel A^{\#}({\lambda }_1))=D_f(A(x_0)\parallel A(x_1))\le \epsilon d(x_0,x_1).$$

$$\mathsf{diam}({\lambda }_0,{\lambda }_1)=\underset{x_0\in \mathsf{supp}({\lambda }_0),x_1\in \mathsf{supp}({\lambda }_1)}{\max }d(x_0,x_1),$$

$$D(A^{\#}(s,\lambda )\parallel A^{\#}(s^{\prime },{\lambda }^{\prime }))\le \epsilon .$$

$$C(s,x)[y]=$$

$$\sum _{x\in \mathcal{X},y\in \mathcal{Y}}\lambda [x]A(x)[y]d(x,y).$$

$$D_f(A^{\#}({\lambda }_0)\parallel A^{\#}({\lambda }_1))$$

$$=$$

$$(x_0,x_1)$$

$$=$$

$$c={\sum }_{x_0,x_1}\gamma [x_0,x_1]A(x_1)[y]$$

$$=$$

$$\le$$

$$f$$

$$=$$

$$D_f(A^{\#}({\lambda }_0)\parallel A^{\#}({\lambda }_1))$$

$$=$$

$$\le$$

$$(x_0,x_1)\in \mathsf{supp}(\gamma )\subseteq \Phi$$

$$=$$

$$D_f(A^{\#}({\lambda }_0)\parallel A^{\#}({\lambda }_1))$$

$$=$$

$$\le$$

$$(x_0,x_1)\in \mathsf{supp}(\gamma )\subseteq \Phi$$

$$=$$

$$D(A(x_0)\parallel A(x_1))$$

$$\le \epsilon W_{p,d}({\eta }_{x_0},{\eta }_{x_1})\text{(by XDistP of A)}$$

$$=\epsilon d(x_0,x_1),$$

$$C^{\#}(s_0,{\lambda }_{s_0})[R]$$

Full text

I Introduction

Differential privacy (DP) [1] is one of the most popular privacy notions that have been studied in various areas, including databases, machine learning, geo-locations, and social networks. The protection of DP can be achieved by adding probabilistic noise to the data we want to obfuscate. In particular, many studies have proposed local obfuscation mechanisms [2, 3, 4] that perturb each single “point” datum (e.g., a geo-location point) by adding controlled probabilistic noise before sending it out to a data collector.

Recent researches [5, 6, 7] show that local obfuscation mechanisms can be used to hide the probability distributions that lie behind such point data and implicitly represent sensitive attributes (e.g., age, gender, social status). In particular, [6] proposes the notion of distribution privacy (DistP) as the local DP of probability distributions. Roughly, DistP of a local obfuscation mechanism $\mathit{A}$ represents that the adversary cannot significantly gain information on the distribution of $\mathit{A}$’s input by observing $\mathit{A}$’s output. However, since DistP assumes the worst case risk in the sense of DP, it imposes strong requirement and might unnecessarily lose the utility of obfuscated data.

In this paper, we relax the notion of DistP by generalizing it to an arbitrary divergence. The basic idea is similar to point privacy notions that relax DP and improve utility by relying on some divergence (e.g., total variation privacy [8], Kullback-Leibler divergence privacy [8, 9], and Rényi differential privacy [10]). We define the notion of divergence distribution privacy by replacing the DP-style with an arbitrary divergence $D$. This relaxation allows us to formalize “on-average” DistP, and to explore privacy notions against an adversary performing the statistical hypothesis test corresponding to the divergence [8].

Furthermore, we propose and investigate local obfuscation mechanisms that provide divergence DistP. Specifically, we consider the following two scenarios:

1. (i)

when we have no idea on the input distributions; 2. (ii)

when we know exact or approximate information on the input distributions (e.g., when we can use public datasets [11, 12] to learn approximate distributions of locations of male/female users if we want to obfuscate the attribute male/female).

For the scenario (i), we clarify how much perturbation noise should be added to provide $f$-divergence DistP when we use an existing mechanism for obfuscating point data. For the scenario (ii), we introduce a local obfuscation mechanism that provides divergence DistP while optimizing the utility of obfuscated data by using the auxiliary information. Here it should be noted that probability coupling techniques are crucial in constructing divergence DistP mechanisms in both the scenarios.

Our contributions. The main contributions are as follows:

• •

We introduce notions of divergence DistP and investigate theoretical properties of distribution obfuscation, especially the relationships between local distribution obfuscation and probability coupling.

• •

We investigate the relationships among various notions of DistP based on $f$-divergences, such as Kullback-Leibler divergence, which models “on-average” risk.

• •

In the scenario (i), we present how much divergence DistP can be achieved by local obfuscation. In particular, by using probability coupling techniques, we prove that perturbation noise should be added proportionally to the Earth mover’s distance between the input distributions that we want to make indistinguishable.

• •

In the scenario (ii), we propose a local obfuscation mechanism, called a (utility-optimal) coupling mechanism, that provides divergence DistP while minimizing utility loss. The construction of the mechanism relies on solving an optimal transportation problem using probability coupling.

• •

We theoretically evaluate the divergence DistP and utility loss of coupling mechanisms that can use exact/approximate knowledge on the input distributions.

Paper organization.   The rest of this paper is organized as follows. Section II presents background knowledge. Section III introduces notions of divergence DistP. Section IV investigates important properties of divergence DistP, and relationships among privacy notions. Section V shows that in the scenario (i), an $f$-privacy mechanism can provide $f$-divergence DistP. Section VI generalizes DistP to use exact/approximate information on the input distribution in the scenario (ii), and proposes a local mechanism for providing DistP while optimizing utility. Section VII discusses related work and Section VIII concludes.

II Preliminaries

In this section we recall some notions of privacy, divergence, and metrics used in this paper.

II-A Notations

Let $\mathbb{R}^{\geq 0}$ be the set of non-negative real numbers, and $[0,1]\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\{r\in\mathbb{R}^{\geq 0}\mid r\leq 1\}$. Let $\varepsilon,\varepsilon_{0},\varepsilon_{1}\in\mathbb{R}^{\geq 0}$, $\delta,\delta_{0},\delta_{1}\in[0,1]$, and $e$ be the base of natural logarithm.

We denote by $|\mathcal{X}|$ the number of elements in a finite set $\mathcal{X}$, and by $\mathbb{D}\mathcal{X}$ the set of all probability distributions over a set $\mathcal{X}$. Given a probability distribution $\lambda$ over a finite set $\mathcal{X}$, the probability of drawing a value $x$ from $\lambda$ is denoted by $\lambda[x]$. For a finite subset $\mathcal{X}^{\prime}\subseteq\mathcal{X}$, we define $\lambda[\mathcal{X}^{\prime}]$ by $\lambda[\mathcal{X}^{\prime}]=\sum_{x^{\prime}\in\mathcal{X}^{\prime}}\lambda[x^{\prime}]$. For a distribution $\lambda$ over a finite set $\mathcal{X}$, its support is $\mathsf{supp}(\lambda)=\{x\in\mathcal{X}\colon\lambda[x]>0\}$.

For a randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ and a set $R\subseteq\mathcal{Y}$, we denote by $\mathit{A}(x)[R]$ the probability that given an input $x$, $\mathit{A}$ outputs one of the elements of $R$. For a randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ and a distribution $\lambda$ over $\mathcal{X}$, we define ${\mathit{A}}^{\#}(\lambda)$ as the probability distribution of the output of $\mathit{A}$. Formally, the lifting of $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ is the function ${\mathit{A}}^{\#}:\mathbb{D}\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ such that for any $R\subseteq\mathcal{Y}$, ${\mathit{A}}^{\#}(\lambda)[R]\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\sum_{x\in\mathcal{X}}\lambda[x]\mathit{A}(x)[R]$.

II-B Differential Privacy

Differential privacy [1] is a notion of privacy guaranteeing that we cannot learn which of two “adjacent” inputs $x$ and $x^{\prime}$ is used to generate an output of a randomized algorithm. This notion is parameterized by a degree $\varepsilon$ of indistinguishability, a ratio $\delta$ of exception, and some adjacency relation $\varPhi$ over a set $\mathcal{X}$ of data. The formal definition is given as follows.

Definition 1** (Differential privacy)**

A randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,\delta)$-differential privacy (DP) w.r.t. an adjacency relation $\varPhi\subseteq\mathcal{X}\times\mathcal{X}$ if for any $(x,x^{\prime})\in\varPhi$ and any $R\subseteq\mathcal{Y}$,

[TABLE]

where the probability is taken over the random choices in $\mathit{A}$.

Then the protection of DP is stronger for smaller $\varepsilon$ and $\delta$.

DP can be achieved by a local obfuscation mechanism or privacy mechanism (illustrated in Fig. 1), namely a randomized algorithm that adds controlled noise probabilistically to given inputs that we want to protect.

II-C Extended Differential Privacy (XDP)

The notion of DP can be relaxed by incorporating a metric $d$ over the set $\mathcal{X}$ of input data. In [13] Chatzikokolakis et al. propose the notion of “$d$-privacy”, an extension of $(\varepsilon,0)$-DP to a metric $d$ on input data. Intuitively, this notion guarantees that when two inputs $x$ and $x^{\prime}$ are closer in terms of $d$, the output distributions are less distinguishable111Compared to DP, XDP provides weaker privacy and higher utility, as it obfuscates closer points. E.g., [14] shows the planar Laplace mechanism [3] (with XDP) adds less noise than the randomized response (with DP). . Here we show the definition of this extended DP equipped with $\delta$.

Definition 2** (Extended differential privacy)**

Let $d:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$ be a metric. We say that a randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,\delta,d)$-extended differential privacy (XDP) if for all $x,x^{\prime}\in\mathcal{X}$ and $R\subseteq\mathcal{Y}$,

[TABLE]

where the probability is taken over the random choices in $\mathit{A}$.

To achieve XDP, obfuscation mechanisms should add noise proportionally to the distance $d(x,x^{\prime})$ between the two inputs $x$ and $x^{\prime}$ that we want to make indistinguishable, hence more noise is require for a larger $d(x,x^{\prime})$.

II-D Distribution Privacy and Extended Distribution Privacy

Distribution privacy (DistP) [6] is a privacy notion that measures how much information on the input distribution is leaked by an output of a randomized algorithm. For example, let $\lambda_{{\it male}}$ (resp. $\lambda_{{\it female}}$) be a (prior) probability distribution of the locations of the male (resp. female) users. When we observe an output of an obfuscation mechanism $\mathit{A}$ and cannot learn whether the input to $\mathit{A}$ is drawn from $\lambda_{{\it male}}$ or $\lambda_{{\it female}}$, then we say that $\mathit{A}$ provides $(\varepsilon,\delta)$-DistP w.r.t. $(\lambda_{{\it male}},\lambda_{{\it female}})$. Formally, DistP is defined as follows.

Definition 3** (Distribution privacy)**

Let $\varepsilon\in\mathbb{R}^{\geq 0}$ and $\delta\in[0,1]$. We say that a randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,\delta)$-distribution privacy (DistP) w.r.t. an adjacency relation $\varPsi\subseteq\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}$ if its lifting ${\mathit{A}}^{\#}:\mathbb{D}\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,\delta)$-DP w.r.t. $\varPsi$, i.e., for all pairs $(\lambda,\lambda^{\prime})\in\varPsi$ and $R\subseteq\mathcal{Y}$,  we have ${\mathit{A}}^{\#}(\lambda)[R]\leq e^{\varepsilon}\cdot{\mathit{A}}^{\#}(\lambda^{\prime})[R]+\delta{.}$

Next we recall an extension [6] of DistP with a metric $\mathit{d}$ as follows. Intuitively, this extended notion guarantees that when two input distributions are closer, then the output distributions must be less distinguishable.

Definition 4** (Extended distribution privacy)**

Let $d:\!(\mathbb{D}\mathcal{X}\allowbreak\times\mathbb{D}\mathcal{X})\rightarrow\mathbb{R}$ be a metric, and $\varPsi\subseteq\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}$. We say that a mechanism $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,d,\delta)$-extended distribution privacy (XDistP) w.r.t. $\varPsi$ if the lifting ${\mathit{A}}^{\#}$ provides $(\varepsilon,d,\delta)$-XDP w.r.t. $\varPsi$, i.e., for all $(\lambda,\lambda^{\prime})\in\varPsi$ and $R\subseteq\mathcal{Y}$, we have ${\mathit{A}}^{\#}(\lambda)[R]\leq e^{\varepsilon d(\lambda,\lambda^{\prime})}\cdot{\mathit{A}}^{\#}(\lambda^{\prime})[R]+\delta{.}$

Analogously to XDP, noise should be added proportionally to the distance $d(\lambda,\lambda^{\prime})$.

II-E Divergence

A divergence over a non-empty set $\mathcal{Y}$ is a function $\mathit{D}(\cdot\!\parallel\!\cdot\!)\!:\allowbreak\mathbb{D}\mathcal{Y}\times\mathbb{D}\mathcal{Y}\rightarrow\mathbb{R}^{\geq 0}$ such that for all $\mu,\mu^{\prime}\in\mathbb{D}\mathcal{Y}$, (i) $\mathit{D}(\mu\parallel\mu^{\prime})\geq 0$ and (ii) $\mathit{D}(\mu\parallel\mu^{\prime})=0$ iff $\mu=\mu^{\prime}$. Note that a divergence may not be symmetric or subadditive. We denote by $\mathsf{Div}(\mathcal{Y})$ the set of all divergences over $\mathcal{Y}$.

Next we recall the notion of (approximate) max divergence, which can be used to define DP.

Definition 5** (Max divergence)**

Let $\delta\in[0,1]$ and $\mu,\mu^{\prime}\in\mathbb{D}\mathcal{Y}$. Then $\delta$-approximate max divergence between $\mu$, $\mu^{\prime}$ is:

[TABLE]

We recall the notion of the $f$-divergences [15]. As shown in Table II-D, many divergence notions (e.g. Kullback-Leibler-divergence [16]) are instances of $f$-divergence.

Definition 6** ($f$-divergence)**

Let $\mathcal{F}$ be the collection of functions defined by:

[TABLE]

Let $\mathcal{Y}$ be a finite set, and $\mu,\mu^{\prime}\in\mathbb{D}\mathcal{Y}$ such that for every $y\in\mathcal{Y}$, $\mu^{\prime}[y]=0$ implies $\mu[y]=0$. Then for an $f\in\mathcal{F}$, the $f$-divergence of $\mu$ from $\mu^{\prime}$ is defined as:

[TABLE]

II-F Probability Coupling

We recall the notion of probability coupling as follows.

Example 1** (Coupling as transformation of distributions)**

Let us consider two distributions $\lambda$ and $\mu$ shown in Fig. 2. A coupling $\gamma$ of $\lambda$ and $\mu$ shows a way of transforming $\lambda$ to $\mu$. For example, $\gamma[2,1]=0.1$ moves from $\lambda[2]$ to $\mu[1]$.

Formally, a coupling is defined as follows.

Definition 7** (Coupling)**

Given $\lambda\in\mathbb{D}\mathcal{X}_{0}$ and $\mu\in\mathbb{D}\mathcal{X}_{1}$, a coupling of $\lambda$ and $\mu$ is a joint distribution $\gamma\in\mathbb{D}(\mathcal{X}_{0}\times\mathcal{X}_{1})$ such that $\lambda$ and $\mu$ are $\gamma$’s marginal distributions, i.e., for each $x_{0}\in\mathcal{X}_{0}$, $\lambda[x_{0}]=\!\sum_{x^{\prime}_{1}\in\mathcal{X}_{1}}\!\gamma[x_{0},x^{\prime}_{1}]$ and for each $x_{1}\in\mathcal{X}_{1}$, $\mu[x_{1}]=\!\sum_{x^{\prime}_{0}\in\mathcal{X}_{0}}\!\gamma[x^{\prime}_{0},x_{1}]$. We denote by $\mathsf{cp}(\lambda,\mu)$ the set of all couplings of $\lambda$ and $\mu$.

II-G $p$-Wasserstein Metric

Then we recall the $p$-Wasserstein metric [17] between two distributions, which is defined using a coupling as follows.

Definition 8** ($p$-Wasserstein metric)**

Let $\mathit{d}$ be a metric over $\mathcal{X}$, and $p\in\mathbb{R}^{\geq 1}\cup\{\infty\}$. The $p$-Wasserstein metric $\mathit{W}_{p,\mathit{d}}$ w.r.t. $\mathit{d}$ is defined by: for any two distributions $\lambda,\mu\in\mathbb{D}\mathcal{X}$,

[TABLE]

$\mathit{W}_{1,\mathit{d}}$ is also called the Earth mover’s distance.

The intuitive meaning of $\mathit{W}_{1,\mathit{d}}(\lambda,\mu)$ is the minimum cost of transportation from $\lambda$ to $\mu$ in transportation theory. As illustrated in Fig. 2, we regard the distribution $\lambda$ (resp. $\mu$) as the set of points where each point $x$ has weight $\lambda[x]$ (resp. $\mu[x]$), and we move some weight in $\lambda$ from a point $x_{0}$ to another $x_{1}$ to construct $\mu$. We represent by $\gamma[x_{0},x_{1}]$ the amount of weight moved from $x_{0}$ to $x_{1}$.222The amount of weight moved from a point $x_{0}$ in $\lambda$ is given by $\lambda[x_{0}]=\sum_{x^{\prime}_{1}\in\mathcal{X}}\gamma[x_{0},x^{\prime}_{1}]$, while the amount moved into $x_{1}$ in $\mu$ is given by $\mu[x_{1}]=\sum_{x^{\prime}_{0}\in\mathcal{X}}\gamma[x^{\prime}_{0},x_{1}]$. Hence $\gamma$ is a coupling of $\lambda$ and $\mu$.

We denote by $\mathit{d}(x_{0},x_{1})$ the cost (i.e., distance) of move from $x_{0}$ to $x_{1}$. Then the minimum cost of the whole transportation is:

[TABLE]

E.g., in Fig. 2, when the cost function $d$ is the Euclid distance over $\mathcal{X}$ (e.g., $d(2,1)=|2-1|=1$), the transportation $\gamma$ achieves the minimum cost $0.1\cdot 1+0.2\cdot 1=0.3$.

Let $\mathit{\Gamma_{\!p,\mathit{d}}}$ the set of all couplings achieving $\mathit{W}_{p,\mathit{d}}$; i.e.,

[TABLE]

Then $\gamma\in\mathit{\Gamma_{\!{\rm 1,\mathit{d}}}}(\lambda,\mu)$ can be efficiently computed by the North-West corner rule [18] when $\mathit{d}$ is submodular 333$\mathit{d}$ is submodular if $\mathit{d}(x_{0},x_{1})+\mathit{d}(x^{\prime}_{0},x^{\prime}_{1})\leq\mathit{d}(x^{\prime}_{0},x_{1})+\mathit{d}(x_{0},x^{\prime}_{1})$..

III Divergence Distribution Privacy

In this section we introduce new definitions of distribution privacy generalized to an arbitrary divergence $D$. The main motivation is to discuss distribution privacy based on $f$-divergences, especially Kullback-Leibler divergence, which models “on-average” risk.

III-A *Divergence DP and Divergence *XDP

To generalize distribution privacy notions, we first present a generalized formulation of point privacy parameterized with a divergence $D$. Intuitively, we say that a randomized algorithm $\mathit{A}$ provides $(\varepsilon,D)$-DP if a divergence $D$ cannot distinguish the input to $\mathit{A}$ by observing an output of $\mathit{A}$.

Definition 9** (Divergence DP w.r.t. adjacency relation)**

For an adjacency relation $\varPhi\subseteq\mathcal{X}\times\mathcal{X}$ and a divergence $D\in\mathsf{Div}(\mathcal{Y})$, we say that a randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,D)$-DP w.r.t. $\varPhi$ if for all $(x,x^{\prime})\!\in\!\varPhi$, we have $\mathit{D}(\mathit{A}(x)\!\parallel\!\mathit{A}(x^{\prime}))\!\leq\!\varepsilon\mbox{ ~{}and~{} }\mathit{D}(\mathit{A}(x^{\prime})\!\parallel\!\mathit{A}(x))\!\leq\!\varepsilon$ .

Note that some instances of divergence DP are known in the literature. In [8], $(\varepsilon,\mathit{D}_{f})$-DP is called $\varepsilon$-$f$-divergence privacy, $(\varepsilon,\mathit{D}_{\rm KL})$-DP (KLP) is called $\varepsilon$-KL-privacy, and $(\varepsilon,\mathit{D}_{\rm TV})$-DP is called $\varepsilon$-total variation privacy. Furthermore, $(\varepsilon,\mathit{D}_{\infty}^{\delta})$-DP is equivalent to $(\varepsilon,\delta)$-DP, since it is known that $(\varepsilon,\delta)$-DP can be defined using the approximate max divergence $\mathit{D}_{\infty}^{\delta}$ as follows:

Proposition 1

A randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,\delta)$-DP w.r.t. $\varPhi\subseteq\mathcal{X}\times\mathcal{X}$ iff for any $(x,x^{\prime})\in\varPhi$,  $\mathit{D}_{\infty}^{\delta}(\mathit{A}(x)\parallel\mathit{A}(x^{\prime}))\leq\varepsilon$ and $\mathit{D}_{\infty}^{\delta}(\mathit{A}(x^{\prime})\parallel\mathit{A}(x))\leq~{}\varepsilon$.

Next we generalize the notion of extended differential privacy (XDP) to an arbitrary divergence $D$ as follows.

Definition 10** (Divergence XDP)**

Let $d:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$ be a metric, $\varPhi\subseteq\mathcal{X}\times\mathcal{X}$, and $D\in\mathsf{Div}(\mathcal{Y})$. We say that a randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,d,D)$-XDP w.r.t. $\varPhi$ if for all $(x,x^{\prime})\in\varPhi$,  $\mathit{D}(\mathit{A}(x)\parallel\mathit{A}(x^{\prime}))\leq\varepsilon d(x,x^{\prime}){.}$

These notions will be used to define (extended) divergence distribution privacy in the next section.

III-B *Divergence DistP and Divergence *XDistP

In this section we generalize the notion of (extended) distribution privacy to an arbitrary divergence $D$. The main aim of generalization is to present theoretical properties of distribution privacy in a more general form, and also to discuss distribution privacy based on the $f$-divergences.

Intuitively, we say that a randomized algorithm $\mathit{A}$ provides $(\varepsilon,D)$-distribution privacy w.r.t. a set $\varPsi$ of pairs of distributions if for each pair $(\lambda_{0},\lambda_{1})\in\varPsi$, a divergence $D$ cannot distinguish which distribution (of $\lambda_{0}$ and $\lambda_{1}$) is used to generate $\mathit{A}$’s input value.

Definition 11** (Divergence DistP)**

Let $D\in\mathsf{Div}(\mathcal{Y})$, and $\varPsi\subseteq\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}$. We say that a randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,D)$-distribution privacy (DistP) w.r.t. $\varPsi$ if the lifting ${\mathit{A}}^{\#}$ provides $(\varepsilon,D)$-DP w.r.t. $\varPsi$, i.e., for all $(\lambda,\lambda^{\prime})\in\varPsi$,

[TABLE]

As with the generalization of DP to $f$-divergence [8],  $D$-DistP expresses privacy against an adversary performing the hypothesis test corresponding to the divergence $D$. When $D$ involves averaging (e.g., $D=\mathit{D}_{\rm KL}$), $D$-DistP formalizes “on-average” privacy, which relaxes the original DistP.

Next we introduce XDistP parameterized with a divergence $D$. Intuitively, XDistP with a divergence $D$ guarantees that when two input distributions $\lambda$ and $\lambda^{\prime}$ are closer (in terms of a metric $d$), then the output distributions ${\mathit{A}}^{\#}(\lambda)$ and ${\mathit{A}}^{\#}(\lambda^{\prime})$ must be less distinguishable (in terms of $D$).

Definition 12** (Divergence XDistP)**

Let $d$ be a metric over $\mathbb{D}\mathcal{X}$, $D\in\mathsf{Div}(\mathcal{Y})$, and $\varPsi\subseteq\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}$. We say that a randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,d,D)$-extended distribution privacy (XDistP) w.r.t. $\varPsi$ if the lifting ${\mathit{A}}^{\#}$ provides $(\varepsilon,d,D)$-XDP w.r.t. $\varPsi$, i.e., for all $(\lambda,\lambda^{\prime})\in\varPsi$,

[TABLE]

IV Properties of Divergence Distribution Privacy

In this section we show useful properties of divergence distribution privacy, such as compositionality and relationships among distribution privacy notions.

IV-A Basic Properties of Divergence Distribution Privacy

In Tables II and III we summarize the results on two kinds of sequential compositions $\mathbin{\odot}$ (Fig. 3(a)) and $\mathbin{\bullet}$ (Fig. 3(b)), post-processing, and pre-processing for divergence DistP and for divergence XDistP, respectively. We present the details and proofs for these results in

Appendices -D, -E, and -F.

The two kinds of composition have been studies in previous work (e.g., [19, 6]). For two mechanisms $\mathit{A}_{0}$ and $\mathit{A}_{1}$, the composition $\mathit{A}_{1}\mathbin{\odot}\mathit{A}_{0}$ means that an identical input value $x$ is given to two DistP mechanisms $\mathit{A}_{0}$ and $\mathit{A}_{1}$, whereas$\mathit{A}_{1}\mathbin{\bullet}\mathit{A}_{0}$ means that independent inputs $x_{b}$ are provided to mechanisms $\mathit{A}_{b}$. Note that this kind of composition is adaptive in the sense that the output of $\mathit{A}_{1}$ can be dependent on that of $\mathit{A}_{0}$. Hence the compositonality does not hold in general for $f$-divergence, whereas we show the compositionality for KL-divergence in Tables II and III. For non-adaptive sequential composition, the compositionality of divergence DistP/XDistP is straightforward from [20], which show the compositionality of popular $f$-divergences, including total variation and Hellinger distance.

As for pre-processing, we use the following definition of stability [6], which is analogous to the stability for DP.

Definition 13** (Stability)**

Let $c\in\mathbb{N}$, $\varPsi\subseteq\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}$, and $\mathit{W}$ be a metric over $\mathbb{D}\mathcal{X}$. A transformation $T:\mathbb{D}\mathcal{X}\rightarrow\mathbb{D}\mathcal{X}$ is $(c,\varPsi)$-stable if for any $(\lambda_{0},\lambda_{1})\in\varPsi$, $T(\lambda_{0})$ can be reached from $T(\lambda_{1})$ at most $c$-steps over $\varPsi$.  Analogously, $T:\mathbb{D}\mathcal{X}\rightarrow\mathbb{D}\mathcal{X}$ is $(c,\mathit{W})$-stable if for any $\lambda_{0},\lambda_{1}\in\mathbb{D}\mathcal{X}$, $\mathit{W}(T(\lambda_{0}),T(\lambda_{1}))\leq c\mathit{W}(\lambda_{0},\lambda_{1})$.

IV-B Relationships among Distribution Privacy Notions

In Fig. 4 we show the summary of the relationships among notions of divergence XDP and divergence XDistP.

See Appendices -B and -G

for details and proofs.

V Local Mechanisms for Divergence Distribution Privacy

In this section we present how much degree of divergence DistP/XDistP can be achieved by local obfuscation. Specifically, we show how $f$-divergence privacy contribute to the obfuscation of probability distributions. To prove those results, we use the notion of probability coupling.

V-A Divergence DistP by Local Obfuscation

We first show that $f$-divergence privacy mechanisms provide $\mathit{D}_{f}$-DistP. To present this formally, we recall the notion of the lifting of relations as follows.

Definition 14** (Lifting of relations)**

Given a relation $\varPhi\subseteq\mathcal{X}\times\mathcal{X}$, the lifting of $\varPhi$ is the maximum relation ${\varPhi}^{\#}\subseteq\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}$ such that for any $(\lambda_{0},\lambda_{1})\in{\varPhi}^{\#}$, there exists a coupling $\gamma\in\mathsf{cp}(\lambda_{0},\lambda_{1})$ satisfying $\mathsf{supp}(\gamma)\subseteq\varPhi$.

Intuitively, when $\lambda_{0}$ and $\lambda_{1}$ are adjacent w.r.t. the lifted relation ${\varPhi}^{\#}$, then we can construct $\lambda_{1}$ from $\lambda_{0}$ according to the coupling $\gamma$, that is, only by moving mass from $\lambda_{0}[x_{0}]$ to $\lambda_{1}[x_{1}]$ where $(x_{0},x_{1})\in\varPhi$ (i.e., $x_{0}$ is adjacent to $x_{1}$). Note that by Definition 7, the coupling $\gamma$ is a probability distribution over $\varPhi$ whose marginal distributions are $\lambda_{0}$ and $\lambda_{1}$. If $\varPhi=\mathcal{X}\times\mathcal{X}$, then ${\varPhi}^{\#}=\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}$.

Now we show that every $f$-divergence privacy mechanism provides $\mathit{D}_{f}$-DistP as follows. (See Appendix -A for the proof.)

Theorem 1** ($(\varepsilon,\mathit{D}_{f})$-DP $\Rightarrow$ $(\varepsilon,\mathit{D}_{f})$-DistP)**

Let $\varPhi\subseteq\mathcal{X}\times\mathcal{X}$. If a randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,\mathit{D}_{f})$-DP w.r.t. $\varPhi$, then it provides $(\varepsilon,\mathit{D}_{f})$-DistP w.r.t. ${\varPhi}^{\#}$.

Intuitively, the $f$-divergence privacy mechanism $\mathit{A}$ makes any pair $(\lambda_{0},\lambda_{1})$ of input distributions in ${\varPhi}^{\#}$ indistinguishable in terms of $\mathit{D}_{f}$ up to the threshold $\varepsilon$.

V-B Divergence XDistP by Local Obfuscation

Next we investigate how much noise should be added for local obfuscation mechanisms to provide divergence XDistP.

We first consider two point distributions $\lambda_{0}$ at $x_{0}$ and $\lambda_{1}$ at $x_{1}$, i.e., $\lambda_{0}[x_{0}]=\lambda_{1}[x_{1}]=1$. Then an $(\varepsilon,\mathit{d},\mathit{D}_{f})$-XDP mechanism $\mathit{A}$ satisfies:

[TABLE]

Hence the noise added by $\mathit{A}$ should be proportional to the distance $\mathit{d}(x_{0},x_{1})$ between $x_{0}$ and $x_{1}$.

To generalize this observation on point distributions to arbitrary distributions, we need to employ some metric between distributions. As the metric, we could use the diameter over the supports, which is defined by:

[TABLE]

or the $\infty$-Wasserstein metric $\mathit{W}_{\infty,\mathit{d}}$, which is used for XDistP [6]. However, when there is an outlier in $\lambda_{0}$ or $\lambda_{1}$, then $\mathsf{diam}(\lambda_{0},\lambda_{1})$ and $\mathit{W}_{\infty,\mathit{d}}(\lambda_{0},\lambda_{1})$ tend to be large. Since the mechanism needs to add noise proportionally to the distance $\mathsf{diam}(\lambda_{0},\lambda_{1})$ or $\mathit{W}_{\infty,\mathit{d}}(\lambda_{0},\lambda_{1})$ to achieve XDistP, it needs to add large amount of noise and thus loses utility significantly.

To have better utility, we employ the Earth mover’s distance ($1$-Wasserstein metric) $\mathit{W}_{1,\mathit{d}}$ as a metric for $\mathit{D}_{f}$-XDistP mechanisms. Given two distributions $\lambda_{0}$ and $\lambda_{1}$ over $\mathcal{X}$, we consider a transportation $\gamma$ from $\lambda_{0}$ to $\lambda_{1}$ that minimizes the expected cost of the transportation. Then the minimum of the expected cost is given by the Earth mover’s distance $\mathit{W}_{1,\mathit{d}}(\lambda_{0},\lambda_{1})$.

Now we show that, to achieve $\mathit{D}_{f}$-XDistP, we only have to add noise proportionally to the Earth mover’s distance $\mathit{W}_{1,\mathit{d}}$ between the input distributions. To formalize this, we define a lifted relation ${\varPhi}_{\mathit{W}_{p}}^{\#}$ as the maximum relation over $\mathbb{D}\mathcal{X}$ s.t. for any $(\lambda_{0},\lambda_{1})\in{\varPhi}_{\mathit{W}_{p}}^{\#}$, there is a coupling $\gamma\in\mathsf{cp}(\lambda_{0},\lambda_{1})$ satisfying $\mathsf{supp}(\gamma)\subseteq\varPhi$ and $\gamma\in\mathit{\Gamma_{\!p,\mathit{d}}}(\lambda_{0},\lambda_{1})$.

Theorem 2** ($(\varepsilon,\mathit{d},\mathit{D}_{f})$-XDP $\Rightarrow$ $(\varepsilon,\mathit{W}_{1,\mathit{d}},\mathit{D}_{f})$-XDistP)**

Let $\mathit{d}:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$ be a metric. If a randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,\mathit{d},\mathit{D}_{f})$-XDP w.r.t. $\varPhi$ then it provides $(\varepsilon,\mathit{W}_{1,\mathit{d}},\mathit{D}_{f})$-XDistP w.r.t. ${\varPhi}_{\mathit{W}_{1}}^{\#}$.

See Appendix -A for the proof. Since the Earth mover’s distance is not grater than the diameter or $\infty$-Wasserstein distance, $\mathit{D}_{f}$-XDistP may require less noise than $\mathit{D}_{\infty}$-XDistP.

VI Local Distribution Obfuscation with Auxiliary Inputs

In this section we introduce a local obfuscation mechanism which we call a coupling mechanism in order to provide distribution privacy while optimizing utility. Specifically, a coupling mechanism uses (full or approximate) knowledge on the input probability distributions to perturb each single input value so that the output distribution gets indistinguishable from some target probability distribution. To define the mechanism, we calculate the probability coupling of each input distribution and the target distribution.

VI-A Privacy Definitions with Auxiliary Inputs

We first extend the definition of divergence DistP so that a local obfuscation mechanism $\mathit{A}$ can receive some auxiliary input (e.g. context information) ranging over a set $\mathcal{S}$, which might be used for $\mathit{A}$ to apply different randomized algorithms in different situations or to different input distributions.

Definition 15** (Divergence DistP with auxiliary inputs)**

Let $\varepsilon\in\mathbb{R}^{\geq 0}$, $D\in\mathsf{Div}(\mathcal{Y})$, and $\varPsi\subseteq(\mathcal{S}\times\mathbb{D}\mathcal{X})\times(\mathcal{S}\times\mathbb{D}\mathcal{X})$. We say that a randomized algorithm $\mathit{A}:\mathcal{S}\times\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,D)$-distribution privacy w.r.t. $\varPsi$ if for all pairs $((s,\lambda),(s^{\prime},\lambda^{\prime}))\in\varPsi$,

[TABLE]

In this definition, the auxiliary input over $\mathcal{S}$ typically represents contextual information about where the obfuscation mechanism $\mathit{A}$ is used or what distribution an input is sampled from. Such information may be useful to customize $\mathit{A}$ to improve utility while providing distribution privacy in specific situations. For example, assume that each auxiliary input $s$ represents the fact that an input $x$ is sampled from a distribution $\lambda_{s}$. If a local mechanism $\mathit{A}$ uses this auxiliary information to always produce a distribution $\mu$ of outputs444If $\mathit{A}$ can use no auxiliary information but wants to produce $\mu$, then the output value needs to be independent of the input, hence very poor utility., it can prevent the leakage of information on the input distribution $\lambda_{s}$. We elaborate on this in the next sections.

VI-B Coupling Mechanisms

In this section we introduce a new local obfuscation mechanism, which we call a coupling mechanism. The aim of the new mechanism is to improve the utility while protecting distribution privacy when we know the input distribution fully or approximately. Intuitively, a coupling mechanism uses (full or partial) information on the input distribution $\lambda\in\mathbb{D}\mathcal{X}$ and produces an output value following some identical distribution $\mu\in\mathbb{D}\mathcal{Y}$, which we call a target distribution. More specifically, given some auxiliary information $s$ about $\lambda$, a coupling mechanism $\mathit{A}:\mathcal{S}\times\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ probabilistically maps each input value $x$ to some output value $y$ so that $y$ is distributed over the target distribution $\mu$.

The simplest construction of a coupling mechanism would be to randomly sample a value $y$ from $\mu$ independently of the input $x$. However, this mechanism provides very poor utility, since the output $y$ loses all information on $x$.

Instead, we construct a mechanism by calculating a coupling $\gamma\in\mathbb{D}(\mathcal{X}\times\mathcal{Y})$ that transforms $\lambda$ to $\mu$ with the minimum loss. We explain this using a simple example below.

Example 2** (Coupling mechanism)**

A coupling $\gamma$ of two distributions $\lambda$ and $\mu$ (Fig. 2(b)) shows a way of transforming $\lambda$ to $\mu$ by probabilistically adding noise to each single input value drawn from $\lambda$. More specifically, $\gamma[2,1]=0.1$ means that $0.1$ (out of $\lambda[2]=0.5$) moves from $2$ to $1$, and $\gamma[2,3]=0.2$ means that $0.2$ moves from $2$ to $3$. Based on this coupling $\gamma$, we construct the coupling mechanism $\mathit{C}$ that maps the input $2$ to the output $1$ with probability $20\%(=0.1/0.5)$, and to the output $3$ with probability $40\%(=0.2/0.5)$. By applying this mechanism $\mathit{C}$ to the input distribution $\lambda$, the resulting output distribution ${\mathit{C}}^{\#}(\lambda)$ is identical to $\mu$.

Formally, we assume that for each auxiliary input $s\in\mathcal{S}$, we learn that the input distribution is approximately $\widehat{\lambda}_{s}\in\mathbb{D}\mathcal{X}$ while the actual distribution is $\lambda_{s}\in\mathbb{D}\mathcal{X}$. Then we define the coupling mechanism $\mathit{C}$ as follows.

Definition 16** (Coupling mechanism)**

Let $\mu\in\mathbb{D}\mathcal{Y}$. For each $s\in\mathcal{S}$, let $\widehat{\lambda}_{s}\in\mathbb{D}\mathcal{X}$ be an approximate input distribution, and $\gamma_{s}\in\mathsf{cp}(\widehat{\lambda}_{s},\mu)$ be a coupling of $\widehat{\lambda}_{s}$ and $\mu$. Then a coupling mechanism w.r.t. $\mu$ is defined as a randomized algorithm $\mathit{C}:\mathcal{S}\times\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ such that given $s\in\mathcal{S}$ and $x\in\mathcal{X}$, outputs $y\in\mathcal{Y}$ with the probability:

[TABLE]

When $\mathit{C}$ can access the exact information on $\lambda_{s}$ (i.e., $\widehat{\lambda}_{s}$ is identical to the actual distribution $\lambda_{s}$ from which inputs are sampled), then $\mathit{C}$ provides $(0,D)$-DistP for any divergence $D$, i.e., no information on the input distribution is leaked by the output of $\mathit{C}$. However, we often obtain only approximate information on the input distribution. In this case, $\mathit{C}$ still provides strong privacy as shown in the next section.

VI-C Distribution Privacy of Coupling Mechanisms

In this section we evaluate the DistP and utility of coupling mechanisms. (See Appendix -C for the proof.)

Theorem 3** (DistP of the coupling mechanism)**

Let $\varPsi\subseteq(\mathcal{S}\times\mathbb{D}\mathcal{X})\times(\mathcal{S}\times\mathbb{D}\mathcal{X})$ such that each element of $\varPsi$ is of the form $(s,\lambda_{s})$ for some $s\in\mathcal{S}$. Let $\mathit{C}$ be a coupling mechanism w.r.t. a target distribution $\mu$. Assume that for each $s\in\mathcal{S}$, the approximate knowledge $\widehat{\lambda}_{s}$ is close to the actual distribution $\lambda_{s}$ in the sense that $\mathit{D}_{\infty}(\widehat{\lambda}_{s}\parallel\lambda_{s})\leq\varepsilon$ and $\mathit{D}_{\infty}(\lambda_{s}\parallel\widehat{\lambda}_{s})\leq\varepsilon$. Then $\mathit{C}$ provides:

$(2\varepsilon,\mathit{D}_{\infty})$-DistP w.r.t. $\varPsi$; 2. 2.

$(2\varepsilon\,e^{\varepsilon},\mathit{D}_{\rm KL})$-DistP w.r.t. $\varPsi$; 3. 3.

$(e^{\varepsilon}f(e^{2\varepsilon}),\mathit{D}_{f})$-DistP w.r.t. $\varPsi$.

This theorem implies that when the mechanism $\mathit{C}$ learns the exact distribution, i.e., $\widehat{\lambda}_{s}=\lambda_{s}$, then by $\varepsilon=0$ it provides $(0,\mathit{D}_{\infty})$-DistP, hence there is no leaked information on the input distributions. For $\varepsilon\approx 0$, we have $\varepsilon\,e^{\varepsilon}\approx\varepsilon(1+\varepsilon)\approx\varepsilon$, hence $\mathit{C}$ provides approximately $(2\varepsilon,\mathit{D}_{\rm KL})$-DistP .

VI-D Utility-Optimal Coupling Mechanisms

In this section we introduce a utility-optimal coupling mechanism. Here we assume that there is some metric $\mathit{d}$ over $\mathcal{X}\cup\mathcal{Y}$. Then the notion of utility loss of a local obfuscation mechanism is defined as follows.

Definition 17** (Expected utility loss)**

Given an input distribution $\lambda\in\mathbb{D}\mathcal{X}$ and a metric $\mathit{d}$ over $\mathcal{X}\cup\mathcal{Y}$, the expected utility loss of a randomized algorithm $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ is:

[TABLE]

The utility loss of a coupling mechanism depends on the choice of the coupling used in the mechanism. Given an Euclid distance $\mathit{d}$ and an input distribution $\widehat{\lambda}_{s}$, the expected utility loss of a coupling mechanism w.r.t. a target distribution $\mu$ using a coupling $\gamma_{s}$ is represented by $\sum_{(x_{0},x_{1})\in\mathsf{supp}(\gamma_{s})}\mathit{d}(x_{0},x_{1})\gamma_{s}[x_{0},x_{1}]$.

Now we define the coupling mechanism that minimizes the expected utility loss as follows.

Definition 18** (Utility-optimal coupling mechanism)**

Let $\mu\in\mathbb{D}\mathcal{Y}$. A utility-optimal coupling mechanism w.r.t. $\mu$ is a coupling mechanism w.r.t. $\mu$ that uses a coupling $\gamma_{s}\in\mathit{\Gamma_{\!{\rm 1,\mathit{d}}}}(\lambda_{s},\mu)$ for each $s\in\mathcal{S}$.

Proposition 2** (Loss of the coupling mechanism)**

For each $s\in\mathcal{S}$, the expected utility loss of a utility-optimal coupling mechanism w.r.t. a target distribution $\mu\in\mathbb{D}\mathcal{Y}$ is given by the Earth mover’s distance $\mathit{W}_{1,\mathit{d}}(\widehat{\lambda}_{s},\mu)$.

The proof is straightforward from the definition of the Earth mover’s distance. Note that as mentioned in Section II-G, the coupling $\gamma_{s}\in\mathit{\Gamma_{\!{\rm 1,\mathit{d}}}}(\lambda_{s},\mu)$ can be efficiently calculated by the North-West corner rule when $\mathit{d}$ is submodular.

Analogously, we could define a coupling mechanism that minimizes the maximum loss by using a coupling $\gamma_{s}\in\mathit{\Gamma_{\!{\rm\infty,\mathit{d}}}}(\lambda_{s},\mu)$ for each $s\in\mathcal{S}$. Then the worst-case utility loss is given by the $\infty$-Wasserstein metric $\mathit{W}_{\infty,\mathit{d}}(\widehat{\lambda}_{s},\mu)$.

VII Related work

Since the seminal work of Dwork [1] on differential privacy (DP), a lot of its variants have been studied to provide different types of privacy guarantees [21]; e.g., $d$-privacy [13], $f$-divergence privacy [20, 8], mutual-information DP [9], concentrated DP [22], Rényi DP [10], Pufferfish privacy [23], Bayesian DP [24], local DP [2], personalized DP [25], and utility-optimized local DP [26]. All of these are intended to protect single input values instead of input distributions.

A few researches have explored the privacy of distributions. Jelasity et al. [5] propose distributional DP to protect the privacy of distribution parameters $\theta$ in a Bayesian style (unlike DP and DistP). Kawamoto et al. [6] propose the DistP notion in a DP style. Geumlek et al. [7] propose profile-based privacy, a variant of DistP that allows the mechanisms to depend on the perfect knowledge of input distributions. However, these studies deal only with the worst-case risk, and neither relax them to the average-case risk (with divergence) nor allow them to use arbitrary auxiliary information (in spite that available information on input distributions is often approximate only).

There have been many studies (e.g., [27]) on the DP of histogram publishing, which is different from DistP as follows. Histogram publishing is a central mechanism that hides a single record $x\in\mathcal{X}$ and outputs an obfuscated histogram, e.g., $\mu\in\mathbb{D}\mathcal{Y}$, whereas a DistP mechanism is a local mechanism that aims at hiding an input distribution $\lambda\in\mathbb{D}\mathcal{X}$ and outputs a single perturbed value $y\in\mathcal{Y}$. As explained in [6], neither of these implies the other.

VIII Conclusion

We introduced the notions of divergence DistP and presented their useful theoretical properties in a general form. By using probability coupling techniques, we presented how much divergence DistP can be achieved by local obfuscation. In particular, we proved that the perturbation noise should be added proportionally to the Earth mover’s distance between the input distributions. We also proposed a local mechanism called a (utility-optimal) coupling mechanism and theoretically evaluated their DistP and utility loss in the presence of (exact or approximate) knowledge on the input distributions.

As for future work, we are planning to develop various kinds of coupling mechanisms for specific applications, such as location privacy.

-A *Local Mechanisms for $\mathit{D}_{f}$-DistP/*XDistP

We first show the proofs for the $\mathit{D}_{f}$-DistP/XDistP achieved by local obfuscation mechanisms.

See 1

Proof:

Let $(\lambda_{0},\lambda_{1})\in{\varPhi}^{\#}$ and $\Gamma\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\mathsf{cp}(\lambda_{0},\lambda_{1})$.

[TABLE]

Assume that $\mathit{A}$ provides $(\varepsilon,\mathit{D}_{f})$-DP w.r.t. $\varPhi$. By Definition 14, there is a coupling $\gamma\in\Gamma$ with $\mathsf{supp}(\gamma)\subseteq\varPhi$. Then:

[TABLE]

Hence $\mathit{A}$ provides $(\varepsilon,\mathit{D}_{f})$-DistP w.r.t. ${\varPhi}^{\#}$. ∎

See 2

Proof:

Assume that $\mathit{A}$ provides $(\varepsilon,\mathit{d},\mathit{D}_{f})$-XDP w.r.t. $\varPhi$. Let $(\lambda_{0},\lambda_{1})\in{\varPhi}_{\mathit{W}_{1}}^{\#}$. By definition, there exists a coupling $\gamma\in\Gamma$ that satisfies $\mathsf{supp}(\gamma)\subseteq\varPhi$ and $\gamma\in\mathit{\Gamma_{\!{\rm 1,\mathit{d}}}}(\lambda_{0},\lambda_{1})$. Then it follows from (1) in the proof for Theorem 1 that:

[TABLE]

Hence $\mathit{A}$ provides $(\varepsilon,\mathit{W}_{1,\mathit{d}},\mathit{D}_{f})$-XDistP w.r.t. ${\varPhi}_{\mathit{W}_{1}}^{\#}$. ∎

-B Point Obfuscation by Distribution Obfuscation

Next we show that divergence DP is an instance of divergence DistP if an adjacency relation includes pairs of point distributions (i.e., distributions having single points with probability $1$).

Lemma 1

Let $p\in\mathbb{R}^{\geq 1}\cup\{\infty\}$ and $\varPhi\subseteq\mathcal{X}\times\mathcal{X}$. For any $(x_{0},x_{1})\in\varPhi$, we have $(\eta_{x_{0}},\eta_{x_{1}})\in{\varPhi}_{\mathit{W}_{p}}^{\#}$.

Theorem 4** (DistP $\Rightarrow$ DP and XDistP $\Rightarrow$ XDP)**

Let $\varepsilon\in\mathbb{R}^{\geq 0}$,  $p\in\mathbb{R}^{\geq 1}\cup\{\infty\}$,  $D\in\mathsf{Div}(\mathcal{Y})$, $\varPhi\subseteq\mathcal{X}\times\mathcal{X}$, and $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ be a randomized algorithm.

If $\mathit{A}$ provides $(\varepsilon,D)$-DistP w.r.t. ${\varPhi}^{\#}$, then it provides $(\varepsilon,D)$-DP w.r.t. $\varPhi$. 2. 2.

If $\mathit{A}$ provides $(\varepsilon,\mathit{W}_{p,\mathit{d}},D)$-XDistP w.r.t. ${\varPhi}_{\mathit{W}_{p}}^{\#}$, then it provides $(\varepsilon,\mathit{d},D)$-XDP w.r.t. $\varPhi$.

Proof:

We show the first claim as follows. Assume that $\mathit{A}$ provides $(\varepsilon,D)$-DistP w.r.t. ${\varPhi}^{\#}$. Let $(x_{0},x_{1})\in\varPhi$, and $\eta_{x_{0}}$ and $\eta_{x_{1}}$ be the point distributions. By Lemma 1 and ${\varPhi}_{\mathit{W}_{p}}^{\#}\subseteq{\varPhi}^{\#}$, we have $(\eta_{x_{0}},\eta_{x_{1}})\in{\varPhi}^{\#}$. By $(\varepsilon,D)$-DistP, we obtain $\mathit{D}(\mathit{A}(x_{0})\!\parallel\!\mathit{A}(x_{1}))=\mathit{D}({\mathit{A}}^{\#}(\eta_{x_{0}})\!\parallel\!{\mathit{A}}^{\#}(\eta_{x_{1}}))\leq\varepsilon$. Hence $\mathit{A}$ provides $(\varepsilon,D)$-DP w.r.t. $\varPhi$.

Next we show the second claim. Assume that $\mathit{A}$ provides $(\varepsilon,\mathit{W}_{p,\mathit{d}},D)$-XDistP w.r.t. ${\varPhi}_{\mathit{W}_{p}}^{\#}$. Let $(x_{0},x_{1})\in\varPhi$, and $\eta_{x_{0}}$ and $\eta_{x_{1}}$ be the point distributions. By Lemma 1, we have $(\eta_{x_{0}},\eta_{x_{1}})\in{\varPhi}_{\mathit{W}_{p}}^{\#}$. Then we obtain:

[TABLE]

where the last equality follows from the definition of $\mathit{W}_{p,\mathit{d}}$. Hence $\mathit{A}$ provides $(\varepsilon,\mathit{d},D)$-XDP w.r.t. $\varPhi$. ∎

-C Privacy and Utility of Coupling Mechanisms

Next, we show the privacy of the coupling mechanisms.

See 3

Proof:

Let $((s_{0},\lambda_{s_{0}}),(s_{1},\lambda_{s_{1}}))\in\varPsi$, and $R\subseteq\mathcal{Y}$. When $\mathit{C}$ is applied to $\lambda_{s_{0}}$ the output distribution is given by:

[TABLE]

When $\mathit{C}$ is applied to $\lambda_{s_{1}}$ the output distribution is:

[TABLE]

Hence $\textstyle\frac{{\mathit{C}}^{\#}(s_{0},\lambda_{0})[R]}{{\mathit{C}}^{\#}(s_{1},\lambda_{1})[R]}\leq e^{2\varepsilon}$. Therefore $\mathit{C}$ provides $(2\varepsilon,\mathit{D}_{\infty})$-DistP w.r.t. $\varPsi$.

Next the KL-divergence is given by:

[TABLE]

Therefore $\mathit{C}$ provides $(2\varepsilon\,e^{\varepsilon},\mathit{D}_{f})$-DistP w.r.t. $\varPsi$.

Finally, the $f$-divergence is given by:

[TABLE]

Therefore $\mathit{C}$ provides $(e^{\varepsilon}f(e^{2\varepsilon}),\mathit{D}_{f})$-DistP w.r.t. $\varPsi$. ∎

-D Sequential Composition $\mathbin{\odot}$ with Shared Input

We first recall the definition of the sequential composition $\mathbin{\odot}$ with shared input (Fig. 3(a)) in previous work.

Definition 19** (Sequential composition $\mathbin{\odot}$)**

Given two randomized algorithms $\mathit{A}_{0}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{0}$ and $\mathit{A}_{1}:\mathcal{Y}_{0}\allowbreak\times\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{1}$, we define the sequential composition of $\mathit{A}_{0}$ and $\mathit{A}_{1}$ as the randomized algorithm $\mathit{A}_{1}\mathbin{\odot}\mathit{A}_{0}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{1}$ such that for any $x\in\mathcal{X}$,  $(\mathit{A}_{1}\mathbin{\odot}\mathit{A}_{0})(x)=\mathit{A}_{1}(\mathit{A}_{0}(x),x))$.

Then we present the compositionality of $\mathit{D}_{\rm KL}$-DistP. Note that since this composition is adaptive, the compositionality does not hold in general for $f$-divergence.

Proposition 3** (Sequential composition $\mathbin{\odot}$ of $\mathit{D}_{\rm KL}$-DistP)**

Let $\varPhi\subseteq\mathcal{X}\times\mathcal{X}$. If $\mathit{A}_{0}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{0}$ provides $(\varepsilon_{0},\mathit{D}_{\rm KL})$-DistP w.r.t. ${\varPhi}^{\#}$ and for each $y_{0}\in\mathcal{Y}_{0}$, $\mathit{A}_{1}(y_{0}):\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{1}$ provides $(\varepsilon_{1},\mathit{D}_{\rm KL})$-DistP w.r.t. ${\varPhi}^{\#}$, the sequential composition $\mathit{A}_{1}\mathbin{\odot}\mathit{A}_{0}$ provides $(\varepsilon_{0}+\varepsilon_{1},\mathit{D}_{\rm KL})$-DistP w.r.t. ${\varPhi}^{\#}$.

Proof:

By Theorem 4 in Appendix -B, $\mathit{A}_{0}$ provides $(\varepsilon_{0},\mathit{D}_{\rm KL})$-DP w.r.t. $\varPhi$, and for each $y_{0}\in\mathcal{Y}_{0}$, $\mathit{A}_{1}(y_{0})$ provides $(\varepsilon_{1},\mathit{D}_{\rm KL})$-DP w.r.t. $\varPhi$. Let $(x,x^{\prime})\in\varPhi$. Then:

[TABLE]

Hence $\mathit{A}_{1}\mathbin{\odot}\mathit{A}_{0}$ provides $(\varepsilon_{0}+\varepsilon_{1},\mathit{D}_{\rm KL})$-DP w.r.t. $\varPhi$. By Theorem 1, $\mathit{A}_{1}\mathbin{\odot}\mathit{A}_{0}$ provides $(\varepsilon_{0}+\varepsilon_{1},\mathit{D}_{\rm KL})$-DistP w.r.t. ${\varPhi}^{\#}$. ∎

Proposition 4** (Sequential composition $\mathbin{\odot}$ of $\mathit{D}_{\rm KL}$-XDistP)**

Let $\mathit{d}$ be a metric over $\mathcal{X}$, and $\varPhi\subseteq\mathcal{X}\times\mathcal{X}$. If $\mathit{A}_{0}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{0}$ provides $(\varepsilon_{0},\mathit{W}_{1,\mathit{d}},\mathit{D}_{\rm KL})$-XDistP w.r.t. ${\varPhi}_{\mathit{W}_{1}}^{\#}$ and for each $y_{0}\in\mathcal{Y}_{0}$, $\mathit{A}_{1}(y_{0}):\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{1}$ provides $(\varepsilon_{1},\mathit{W}_{1,\mathit{d}},\mathit{D}_{\rm KL})$-XDistP w.r.t. ${\varPhi}_{\mathit{W}_{1}}^{\#}$ then the sequential composition $\mathit{A}_{1}\mathbin{\odot}\mathit{A}_{0}$ provides $(\varepsilon_{0}+\varepsilon_{1},\mathit{W}_{1,\mathit{d}},\allowbreak\mathit{D}_{\rm KL})$-XDistP w.r.t. ${\varPhi}_{\mathit{W}_{1}}^{\#}$.

Proof:

Analogous to the proof for Proposition 3. ∎

-E Sequential Composition $\mathbin{\bullet}$ with Independent Sampling

In this section we present the compositionality with independent sampling, which is defined as follows.

Definition 20** (Sequential composition $\mathbin{\bullet}$)**

Given two randomized algorithms $\mathit{A}_{0}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{0}$ and $\mathit{A}_{1}:\mathcal{Y}_{0}\times\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{1}$, we define the sequential composition of $\mathit{A}_{0}$ and $\mathit{A}_{1}$ as the randomized algorithm $\mathit{A}_{1}\mathbin{\bullet}\mathit{A}_{0}:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{1}$ such that: for any $x_{0},x_{1}\in\mathcal{X}$,  $(\mathit{A}_{1}\mathbin{\bullet}\mathit{A}_{0})(x_{0},x_{1})=\mathit{A}_{1}(\mathit{A}_{0}(x_{0}),x_{1}))$.

We define an operator $\diamond$ between binary relations $\varPsi_{0}$ and $\varPsi_{1}$:

[TABLE]

Now we show the compositionality for $\mathit{D}_{\rm KL}$-DistP.

Proposition 5** (Sequential composition $\mathbin{\bullet}$ of $\mathit{D}_{\rm KL}$-DistP)**

Let $\varPsi\subseteq\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}$. If $\mathit{A}_{0}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{0}$ provides $(\varepsilon_{0},\mathit{D}_{\rm KL})$-DistP w.r.t. $\varPsi$ and for each $y_{0}\in\mathcal{Y}_{0}$, $\mathit{A}_{1}(y_{0}):\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{1}$ provides $(\varepsilon_{1},\mathit{D}_{\rm KL})$-DistP w.r.t. $\varPsi$, then the composition $\mathit{A}_{1}\mathbin{\bullet}\mathit{A}_{0}$ provides $(\varepsilon_{0}+\varepsilon_{1},\mathit{D}_{\rm KL})$-DistP w.r.t. $\varPsi\diamond\varPsi$.

Proof:

Let $(\lambda_{0},\lambda^{\prime}_{0}),(\lambda_{1},\lambda^{\prime}_{1})\in\varPsi$.

[TABLE]

Hence $\mathit{A}_{1}\mathbin{\bullet}\mathit{A}_{0}$ provides $(\varepsilon_{0}+\varepsilon_{1},\mathit{D}_{\rm KL})$-DistP w.r.t. $\varPsi\diamond\varPsi$. ∎

Proposition 6** (Sequential composition $\mathbin{\bullet}$ of $\mathit{D}_{\rm KL}$-XDistP)**

Let $\mathit{d}$ be a metric over $\mathcal{X}$, and $\varPsi\subseteq\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}$. If $\mathit{A}_{0}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{0}$ provides $(\varepsilon_{0},\mathit{W}_{1,\mathit{d}},\mathit{D}_{\rm KL})$-XDistP w.r.t. $\varPsi$ and for each $y_{0}\in\mathcal{Y}_{0}$, $\mathit{A}_{1}(y_{0}):\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}_{1}$ provides $(\varepsilon_{1},\mathit{W}_{1,\mathit{d}},\mathit{D}_{\rm KL})$-XDistP w.r.t. $\varPsi$, then the composition $\mathit{A}_{1}\mathbin{\bullet}\mathit{A}_{0}$ provides $(\varepsilon_{0}+\varepsilon_{1},\mathit{W}_{1,\mathit{d}},\mathit{D}_{\rm KL})$-XDistP w.r.t. $\varPsi\diamond\varPsi$.

Proof:

Analogous to the proof for Proposition 5. ∎

-F Post-processing and Pre-processing

Next we show that divergence distribution privacy is immune to the post-processing. For $\mathit{A}_{0}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ and $\mathit{A}_{1}:\mathcal{Y}\rightarrow\mathbb{D}\mathcal{Z}$, we define $\mathit{A}_{1}\circ\mathit{A}_{0}$ by: $(\mathit{A}_{1}\circ\mathit{A}_{0})(x)=\mathit{A}_{1}(\mathit{A}_{0}(x))$.

Proposition 7** (Post-processing)**

Let $\varPsi\subseteq\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}$, and $\mathit{W}:\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}\rightarrow\mathbb{R}^{\geq 0}$ be a metric. Let $\mathit{A}_{0}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ and $\mathit{A}_{1}:\mathcal{Y}\rightarrow\mathbb{D}\mathcal{Z}$.

If $\mathit{A}_{0}$ provides $(\varepsilon,\mathit{D}_{f})$-DistP w.r.t. $\varPsi$ then so does the composite function $\mathit{A}_{1}\circ\mathit{A}_{0}$. 2. 2.

If $\mathit{A}_{0}$ provides $(\varepsilon,\mathit{W},\mathit{D}_{f})$-XDistP w.r.t. $\varPsi$ then so does the composite function $\mathit{A}_{1}\circ\mathit{A}_{0}$.

Proof:

The claim is immediate from the data processing inequality for the $f$-divergence. ∎

We then show properties of pre-processing as follows.

Proposition 8** (Pre-processing)**

Let $c\in\mathbb{R}^{\geq 0}$, $\varPsi\subseteq\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}$, $\mathit{W}:\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X}\rightarrow\mathbb{R}^{\geq 0}$ be a metric, and $D\in\mathsf{Div}(\mathcal{Y})$.

If $T:\mathbb{D}\mathcal{X}\rightarrow\mathbb{D}\mathcal{X}$ is a $(c,\varPsi)$-stable transformation and $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,D)$-DistP w.r.t. $\varPsi$, then $\mathit{A}\circ T$ provides $(c\,\varepsilon,D)$-DistP w.r.t. $\varPsi$. 2. 2.

If $T:\mathbb{D}\mathcal{X}\rightarrow\mathbb{D}\mathcal{X}$ is a $(c,\mathit{W})$-stable transformation and $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,\mathit{W},D)$-XDistP, then $\mathit{A}\circ T$ provides $(c\,\varepsilon,\mathit{W},D)$-XDistP.

Proof:

We show the first claim as follows. Assume that $\mathit{A}$ provides $(\varepsilon,D)$-DistP w.r.t. $\varPsi$. Let $(\lambda,\lambda^{\prime})\in\varPsi$. Then $\mathit{D}({(\mathit{A}\circ T)}^{\#}(\lambda)\!\parallel\!{(\mathit{A}\circ T)}^{\#}(\lambda^{\prime}))=\mathit{D}({\mathit{A}}^{\#}({T}^{\#}(\lambda))\!\parallel\!{\mathit{A}}^{\#}({T}^{\#}(\lambda^{\prime})))\leq c\varepsilon$ by $(c,\varPsi)$-stability. Therefore $\mathit{A}\circ T$ provides $(c\,\varepsilon,D)$-DistP w.r.t. $\varPsi$.

Next we show the second claim. Assume that $\mathit{A}$ provides $(\varepsilon,\mathit{W},D)$-XDistP. Let $\lambda,\lambda^{\prime}\in\mathbb{D}\mathcal{X}$. Then we obtain:

[TABLE]

Therefore $\mathit{A}\circ T$ provides $(c\,\varepsilon,\mathit{W},D)$-XDistP. ∎

-G Relationships among XDistP Notions

Finally, we show relationships among distribution privacy notions with different metric $d$ and divergence $D$.

Proposition 9** ($\mathit{W}_{1,\mathit{d}}$-XDistP  $\Rightarrow$ $\mathit{W}_{\infty,\mathit{d}}$-XDistP)**

Let $D\in\mathsf{Div}(\mathcal{Y})$. If $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,\mathit{W}_{1,\mathit{d}},\mathit{D})$-XDistP, then it provides $(\varepsilon,\mathit{W}_{\infty,\mathit{d}},\mathit{D})$-XDistP.

Proof:

Assume that $\mathit{A}$ provides $(\varepsilon,\mathit{W}_{1,\mathit{d}},\mathit{D})$-XDistP. Let $\lambda_{0},\lambda_{1}\in\mathbb{D}\mathcal{X}$. By the property of the $p$-Wasserstein metric, $\mathit{W}_{1,\mathit{d}}(\lambda_{0},\lambda_{1})\leq\mathit{W}_{\infty,\mathit{d}}(\lambda_{0},\lambda_{1})$. Then $\mathit{D}(\mu_{0}\parallel\mu_{1})\leq\mathit{W}_{1,\mathit{d}}(\lambda_{0},\lambda_{1})\leq\mathit{W}_{\infty,\mathit{d}}(\lambda_{0},\lambda_{1})$. Hence the claim follows. ∎

Proposition 10** ($\mathit{D}\leq\mathit{D^{\prime}}$ & $\mathit{D^{\prime}}$-XDistP  $\Rightarrow$ $\mathit{D}$-XDistP)**

Let $d:(\mathbb{D}\mathcal{X}\times\mathbb{D}\mathcal{X})\rightarrow\mathbb{R}$ be a metric. Let $D,D^{\prime}\in\mathsf{Div}(\mathcal{Y})$ be two divergences such that for all $\mu_{0},\mu_{1}\in\mathbb{D}\mathcal{Y}$,  $\mathit{D}(\mu_{0}\parallel\mu_{1})\leq\mathit{D^{\prime}}(\mu_{0}\parallel\mu_{1})$. If $\mathit{A}:\mathcal{X}\rightarrow\mathbb{D}\mathcal{Y}$ provides $(\varepsilon,d,\mathit{D^{\prime}})$-XDistP, then it provides $(\varepsilon,d,\mathit{D})$-XDistP.

Then $(\varepsilon,d,\mathit{D}_{\infty})$-XDistP implies $(\varepsilon,d,\mathit{D}_{\rm KL})$-XDistP.

Proof:

Assume $\mathit{A}$ provides $(\varepsilon,d,\mathit{D^{\prime}})$-XDistP. Let $\lambda_{0},\lambda_{1}\in\mathbb{D}\mathcal{X}$. Then $\mathit{D^{\prime}}({\mathit{A}}^{\#}(\lambda_{0}){\parallel}{\mathit{A}}^{\#}(\lambda_{1}))\leq\varepsilon d(\lambda_{0},\lambda_{1}){.}$ By definition, $\mathit{D}({\mathit{A}}^{\#}(\lambda_{0}){\parallel}{\mathit{A}}^{\#}(\lambda_{1}))\leq\mathit{D^{\prime}}({\mathit{A}}^{\#}(\lambda_{0}){\parallel}{\mathit{A}}^{\#}(\lambda_{1}))\allowbreak\leq\varepsilon d(\lambda_{0},\lambda_{1})$. Thus $\mathit{A}$ provides $(\varepsilon,d,\mathit{D})$-XDistP. ∎