Relational $\star$-Liftings for Differential Privacy

TL;DR

This paper introduces a new existential approximate lifting called $ ext{ extsterling}$-lifting, unifying existing notions and enhancing the tools for proving differential privacy through cleaner properties and more general constructions.

Contribution

It proposes a novel $ ext{ extsterling}$-lifting that unifies existing approximate liftings, providing more precise composition theorems and cleaner properties for differential privacy proofs.

Findings

$ ext{ extsterling}$-lifting is equivalent to Sato's universal construction for discrete measures

Unifies all known notions of approximate lifting

Enables richer and more precise differential privacy proofs

Abstract

Recent developments in formal verification have identified approximate liftings (also known as approximate couplings) as a clean, compositional abstraction for proving differential privacy. This construction can be defined in two styles. Earlier definitions require the existence of one or more witness distributions, while a recent definition by Sato uses universal quantification over all sets of samples. These notions have each have their own strengths: the universal version is more general than the existential ones, while existential liftings are known to satisfy more precise composition principles. We propose a novel, existential version of approximate lifting, called $\star$-lifting, and show that it is equivalent to Sato's construction for discrete probability measures. Our work unifies all known notions of approximate lifting, yielding cleaner properties, more general constructions, and more precise composition theorems for both styles of lifting, enabling richer proofs of differential privacy. We also clarify the relation between existing definitions of approximate lifting, and consider more general approximate liftings based on $f$-divergences.

Figures

Equations

$${\mathbb{P}}_{M(a)}[E]\le e^{\epsilon }\cdot {\mathbb{P}}_{M(a^{\prime })}[E]+\delta .$$

$$\underset{E\subseteq B}{\sup }\left({\mathbb{P}}_{{\mu }_1}[E]-e^{\epsilon }\cdot {\mathbb{P}}_{{\mu }_2}[E]\right).$$

$$f_1^{♯}({\mu }_1){\mathcal{R}}_{\epsilon ,\delta }^{(2)}{f}_2^{♯}({\mu }_2)⟺{\mu }_1{\mathcal{S}}_{\epsilon ,\delta }^{(2)}{\mu }_2$$

$${\mathbb{P}}_{\mu }[P_2]\le e^{\epsilon }\cdot {\mathbb{P}}_{\mu }[P_1]⟺\mu {\mathcal{R}}_{\epsilon ,0}^{(2)}\mu ,$$

$$\begin{array}{c}{\eta }_{\text{mspace}-1.5mu⊲}(a,b)=\{\begin{array}{ll}{\mu }_{\text{mspace}-1.5mu⊲}(a,b)& :(a,b)\in S_1\times S_2\\ {\mu }_{\text{mspace}-1.5mu⊲}[a,B^{\star }-S_2]& :b=\star \end{array}\\ {\eta }_{\text{mspace}-1.5mu⊳}(a,b)=\{\begin{array}{ll}{\mu }_{\text{mspace}-1.5mu⊳}(a,b)& :(a,b)\in S_1\times S_2\\ {\mu }_{\text{mspace}-1.5mu⊳}[A^{\star }-S_1,b]& :a=\star \end{array}\end{array}$$

$$\zeta (a,b)=\{\begin{array}{ll}\delta (a,b)& :(a,b)\in S_1\times S_2\cup \{\star \}\times B\\ \delta [a,B^{\star }-S_2]& :b=\star .\end{array}$$

$$\forall a_1,a_2\in A,\forall X\subseteq B.(a_1,a_2)\in Phys.Rev.\mathrm{E}⟹P(a_1)[X]\le e^{\epsilon }\cdot P(a_2)[X]+\delta .$$

$${\mathbb{E}}_{{\mu }_1}[{\eta }_1]{\mathcal{S}}_{\epsilon +{\epsilon }^{\prime },\delta +{\delta }^{\prime }}^{(\star )}{\mathbb{E}}_{{\mu }_2}[{\eta }_2].$$

$${\mu }_1[X]$$

$$\le e^{\epsilon }\cdot (e^{{\epsilon }^{\prime }}\cdot {\mu }_3[\mathcal{S}(\mathcal{R}(X))]+{\delta }^{\prime })+\delta$$

$$=e^{\epsilon +{\epsilon }^{\prime }}\cdot {\mu }_3[\mathcal{S}(\mathcal{R}(X))]+e^{\epsilon }\cdot {\delta }^{\prime }+\delta .$$

$${\eta }_{\text{mspace}-1.5mu⊲}:a$$

$$⟨{\xi }_{\text{mspace}-1.5mu⊲},{\xi }_{\text{mspace}-1.5mu⊳}⟩{◀}_{\epsilon +{\epsilon }^{\prime },\delta +{\delta }^{\prime }}^{\mathcal{S}}⟨{\mathbb{E}}_{{\mu }_1}[{\eta }_1]\&{\mathbb{E}}_{{\mu }_2}[{\eta }_2]⟩.$$

$$\forall X\subseteq A.{\mu }_1[X]\le e^{\epsilon }\cdot {\mu }_2[\mathcal{R}(X)]+\delta .$$

$$\forall X\subseteq A.{\mu }_1[X]\le {\mu }_2[\mathcal{R}(X)].$$

$$\sup \{|f|\mid \text{f is a flow for N}\}=\inf \{|C|\mid \text{C is a cut for N}\}$$

$$E_{\top }$$

$$E_{\mathcal{R}}$$

$$|C|$$

$$\ge e^{-\epsilon }\cdot {\mu }_1[A^{†}]+{\mu }_2[\mathcal{R}(A-A^{†})]+e^{-\epsilon }\cdot \delta +(\omega -e^{-\epsilon }\cdot |{\mu }_1|)$$

$$\ge e^{-\epsilon }\cdot ({\mu }_1[A^{†}]+{\mu }_1[A-A^{†}])+\omega -e^{-\epsilon }\cdot |{\mu }_1|=\omega .$$

$$\begin{array}[]{ll}\begin{aligned} \eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}:A\times B^{\star}&\to\mathbb{R}^{\geq 0}\\ (a,b)&\mapsto e^{\varepsilon}\cdot\hat{f}(a,b)\end{aligned}&\begin{aligned} \eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}:A^{\star}\times B&\to\mathbb{R}^{\geq 0}\\ (a,b)&\mapsto\hat{f}(a,b).\end{aligned}\end{array}$$

$$\overline{{\eta }_{\text{mspace}-1.5mu⊲}}[X]-e^{\epsilon }\cdot \overline{{\eta }_{\text{mspace}-1.5mu⊳}}[X]$$

$$\le e^{\epsilon }\cdot \hat{f}[\overline{X_a}\times \{\star \}]\le e^{\epsilon }\cdot \hat{f}[A\times \{\star \}]=\delta .$$

$${\mu }_1[X]$$

$$=\overline{{\eta }_{\text{mspace}-1.5mu⊲}}[X,\mathcal{R}(X)\uplus \{\star \}]+\underset{=0}{\underbrace{\overline{{\eta }_{\text{mspace}-1.5mu⊲}}[X,\overline{\mathcal{R}(X)}]}}\le e^{\epsilon }\cdot \overline{{\eta }_{\text{mspace}-1.5mu⊳}}[X,\mathcal{R}(X)\uplus \{\star \}]+\delta$$

$$\le e^{\epsilon }\cdot \underset{={\eta }_{\text{mspace}-1.5mu⊳}[A^{\star },\mathcal{R}(X)]}{\underbrace{\overline{{\eta }_{\text{mspace}-1.5mu⊳}}[A^{\star },\mathcal{R}(X)]}}+e^{\epsilon }\cdot \underset{=0}{\underbrace{\overline{{\eta }_{\text{mspace}-1.5mu⊳}}[A^{\star },\{\star \}]}}+\delta$$

$$=e^{\epsilon }\cdot {\pi }_2^{♯}({\eta }_{\text{mspace}-1.5mu⊳})[\mathcal{R}(X)]+\delta =e^{\epsilon }\cdot {\mu }_2[\mathcal{R}(X)]+\delta ,$$

$${\eta }_{\text{mspace}-1.5mu⊳}(a,b)\le {\eta }_{\text{mspace}-1.5mu⊲}(a,b)\le e^{\epsilon }\cdot {\eta }_{\text{mspace}-1.5mu⊳}(a,b).$$

$${\nu }_{\text{mspace}-1.5mu⊲}(a,b)$$

$${\nu }_{\text{mspace}-1.5mu⊳}(a,b)$$

$${\eta }_{\text{mspace}-1.5mu⊲}$$

$${\eta }_{\text{mspace}-1.5mu⊳}$$

$${\eta }_{\text{mspace}-1.5mu⊲}(a,\star )$$

$$\ge {\mu }_1(a)-\sum _{b\in B^{\star }}{\mu }_{\text{mspace}-1.5mu⊲}(a,b)={\mu }_1(a)-{\mu }_1(a)=0,$$

$${\eta }_{\text{mspace}-1.5mu⊳}(\star ,b)$$

$$\ge {\mu }_2(b)-\sum _{a\in A^{\star }}{\mu }_{\text{mspace}-1.5mu⊳}(a,b)={\mu }_2(b)-{\mu }_2(b)=0.$$

Full text

We propose a novel, existential version of approximate lifting, called $\star$-lifting, and show that it is equivalent to Sato’s construction for discrete probability measures. Our work unifies all known notions of approximate lifting, yielding cleaner properties, more general constructions, and more precise composition theorems for both styles of lifting, enabling richer proofs of differential privacy. We also clarify the relation between existing definitions of approximate lifting, and consider more general approximate liftings based on $f$-divergences.

1. Introduction

Differential privacy [Dwork et al., 2006] is a strong, rigorous notion of statistical privacy. Informally, differential privacy guarantees to every individual that their participation in a database query will have a quantitatively small effect on the query results, limiting the amount that the query answer depends on their private data. The definition of differential privacy is parametrized by two non-negative real numbers, $(\varepsilon,\delta)$, which quantify the effect of individuals on the output of the private query: smaller values give stronger privacy guarantees. The main strengths of differential privacy lie in its theoretical elegance, minimal assumptions, and flexibility.

Recently, programming language researchers have developed approaches based on dynamic analysis, type systems, and program logics for formally proving differential privacy for programs. (We refer the interested reader to a recent survey [Barthe et al., 2016c] for an overview of this growing field.) In this paper, we consider approaches based on relational program logics [Barthe et al., 2013, Barthe and Olmedo, 2013, Olmedo, 2014, Barthe et al., 2016a, b, Sato, 2016]. To capture the quantitative nature of differential privacy, these systems rely on a quantitative generalization of probabilistic couplings (see, e.g., [Lindvall, 2002, Thorisson, 2000, Villani, 2008]), called approximate liftings or $(\varepsilon,\delta)$-liftings. Prior works have considered several potential definitions. While all definitions support compositional reasoning and enable program logics that can verify complex examples from the privacy literature, the various notions of approximate liftings have different strengths and weaknesses.

Broadly speaking, the first class of definitions require the existence of one or two witness distributions that “couple” the output distributions from program executions on two related inputs (intuitively, the true database and the true database omitting one individual’s record). The earliest definition [Barthe et al., 2013] supports accuracy-based reasoning for the Laplace mechanism, while subsequent definitions [Barthe and Olmedo, 2013, Olmedo, 2014] support more precise composition principles from differential privacy and can be generalized to other notions of distance on distributions. These definitions, and their associated program logics, were designed for discrete distributions.

In the course of extending these ideas to continuous distributions, Sato [2016] proposes a radically different notion of approximate lifting that does not rely on witness distributions. Instead, it uses a universal quantification over all sets of samples. Sato shows that this definition is strictly more general than the existential versions, but it is unclear (a) whether the gap can be closed and (b) whether his construction satisfies the same composition principles enjoyed by some existential definitions.

As a consequence, no single definition is known to satisfy the properties needed to support all existing formalized proofs of differential privacy. Furthermore, some of the most involved privacy proofs cannot be formalized at all, as their proofs require a combination of constructions satisfied by existential or universal liftings, but not both.

Outline of the paper

After introducing mathematical preliminaries in Section 2, we introduce our main technical contribution: a new, existential definition of approximate lifting. This construction, which we call $\star$-lifting, is a generalization of an existing definition by Barthe and Olmedo [2013], Olmedo [2014]. The key idea is to slightly enlarge the domain of witness distributions with a single generic point, broadening the class of approximate liftings. By a maximum flow/minimum cut argument, we show that $\star$-liftings are equivalent to Sato’s lifting over discrete distributions. This equivalence can be viewed as an approximate version of Strassen’s theorem [Strassen, 1965], a classical result in probability theory characterizing the existence of probabilistic couplings. We present our definition and the proof of equivalence in Section 3.

Then, we show that $\star$-liftings satisfy desirable theoretical properties by leveraging the equivalence of liftings in two ways. In one direction, Sato’s definition gives simpler proofs of more general properties of $\star$-liftings. In the other direction, $\star$-liftings—like previously proposed existential liftings—can smoothly incorporate composition principles from the theory of differential privacy. In particular, our connection shows that Sato’s definition can use these principles in the discrete case. We describe the key theoretical properties of $\star$-liftings in Section 4.

Finally, we provide a thorough comparison of $\star$-lifting with other existing definitions of approximate lifting in Section 5, introduce a symmetric version of $\star$-lifting that satisfies the so-called advanced composition theorem from differential privacy Dwork et al. [2010] in Section 6, and generalize $\star$-liftings to approximate liftings based on $f$-divergences in Section 7.

Overall, the equivalence of $\star$-liftings and Sato’s lifting, along with the natural theoretical properties satisfied by the common notion, suggest that these definitions are two views on the same concept: an approximate version of probabilistic coupling.

2. Background

To model probabilistic data, we work with discrete sub-distributions.

{defi}

A sub-distribution over a set $A$ is defined by its mass function $\mu:A\to[0,1]$, which gives the probability of the singleton events $a\in A$. This mass function must be s.t. $|{\mu}|\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\sum_{a\in A}\mu(a)$ is well-defined and at most $1$. In particular, the support $\operatorname{supp}(\mu)\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\{a\in A\mid\mu(a)\neq 0\}$ must be discrete (i.e. finite or countably infinite). When the weight $|{\mu}|$ is equal to $1$, we call $\mu$ a (proper) distribution. We let ${\mathbb{D}}(A)$ denote the set of sub-distributions over $A$. Events $E$ are predicates on $A$; the probability of an event $E(x)$ w.r.t. $\mu$, written $\mathbb{P}_{{x}\sim{\mu}}[{E(x)}]$ or $\mathbb{P}_{\mu}[{E}]$, is defined as $\sum_{x\in A\mid E(x)}\mu(x)$.

Simple examples of sub-distributions include the null sub-distribution ${{\mathbb{0}}^{A}}\in{\mathbb{D}}(A)$, which maps each element of $A$ to [math], and the Dirac distribution centered on $x$, written ${{\mathbb{1}}_{x}}$, which maps $x$ to $1$ and all other elements to [math]. One can equip distributions with the usual monadic structure using the Dirac distributions ${{\mathbb{1}}_{x}}$ for the unit and distribution expectation $\mathbb{E}_{{x}\sim{\mu}}[{f(x)}]$ for the bind; if $\mu$ is a distribution over $A$ and $f$ has type $A\to{\mathbb{D}}(B)$, then the bind defines a sub-distribution over $B$ via $\mathbb{E}_{{a}\sim{\mu}}[{f(a)}]:b\mapsto\sum_{a}\mu(a)\cdot f(a)(b)$.

If $f:A\to B$, we can lift $f$ to a function $f^{\sharp}:{\mathbb{D}}(A)\to{\mathbb{D}}(B)$ as $f^{\sharp}(\mu)\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\mathbb{E}_{{a}\sim{\mu}}[{{{\mathbb{1}}_{f(a)}}}]$ ; more explicitly, $f^{\sharp}(\mu):b\mapsto\mathbb{P}_{{a}\sim{\mu}}[{a\in f^{-1}(b)}]$. For instance, when working with sub-distributions over pairs, we have the probabilistic versions $\pi_{1}^{\sharp}:{\mathbb{D}}(A\times B)\to{\mathbb{D}}(A)$ and $\pi_{2}^{\sharp}:{\mathbb{D}}(A\times B)\to{\mathbb{D}}(B)$ (called the marginals) of the usual projections $\pi_{1}$ and $\pi_{2}$ by lifting. One can check that the first and second marginals $\pi_{1}^{\sharp}(\mu)$ and $\pi_{2}^{\sharp}(\mu)$ of a distribution $\mu$ over $A\times B$ are given by the following equations: $\pi_{1}^{\sharp}(\mu)(a)=\sum_{b\in B}\mu(a,b)$ and $\pi_{2}^{\sharp}(\mu)(b)=\sum_{a\in A}\mu(a,b)$. When $f:A\to{\mathbb{D}}(B)$, we will abuse notation and write the lifting $f^{\sharp}:{\mathbb{D}}(A)\to{\mathbb{D}}(B)$ to mean $f^{\sharp}(\mu)\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\mathbb{E}_{{x}\sim{\mu}}[{f(x)}]$; this is sometimes called the Kleisli extension of $f$.

Finally, we will often consider sums of weight functions over sets. If $\alpha:A\to\mathbb{R}^{\geq 0}$ maps $A$ to the non-negative real numbers, we write $\alpha[X]\in\mathbb{R}^{\geq 0}\cup\{\infty\}$ for $\sum_{x\in X}\alpha(x)$. Moreover, if $\alpha:A\times B\to\mathbb{R}^{\geq 0}$, we write $\alpha[X,Y]$ (resp. $\alpha[x,Y]$, $\alpha[X,y]$) for $\alpha[X\times Y]$ (resp. $\alpha[\{x\}\times Y$, $\alpha[X\times\{y\}]$). Note that for a sub-distribution $\mu\in{\mathbb{D}}(A)$ and an event $E\subseteq A$, $\mathbb{P}_{\mu}[{E}]=\mu[E]$.

We now review the definition of differential privacy.

{defi}

[Dwork et al. [2006]] Let $\varepsilon,\delta\geq 0$ be real parameters. A probabilistic computation $M:A\to{\mathbb{D}}(B)$ satisfies $(\varepsilon,\delta)$-differential privacy w.r.t. an adjacency relation ${\rm Phys.~{}Rev.~{}E}\subseteq A\times A$ if for every pair of inputs $(a,a^{\prime})\in{\rm Phys.~{}Rev.~{}E}$ and every subset of outputs $E\subseteq B$, we have

[TABLE]

Differential privacy is closely related to a relaxed version of distance—technically, an $f$-divergence—on distributions.

{defi}

[Barthe et al. [2013], Barthe and Olmedo [2013], Olmedo [2014]] Let $\varepsilon\geq 0$. The $\varepsilon$-DP divergence $\Delta_{\varepsilon}(\mu_{1},\mu_{2})$ between two sub-distributions $\mu_{1},\mu_{2}\in{\mathbb{D}}(B)$ is defined as

[TABLE]

Then, differential privacy admits an alternative characterization based on DP divergence.

Lemma 1**.**

A probabilistic computation $M:A\to{\mathbb{D}}(B)$ satisfies $(\varepsilon,\delta)$-differential privacy w.r.t. an adjacency relation ${\rm Phys.~{}Rev.~{}E}\subseteq A\times A$ iff $\Delta_{\varepsilon}(M(a),M(a^{\prime}))\leq\delta$ for every pair of inputs $(a,a^{\prime})\in{\rm Phys.~{}Rev.~{}E}$.

Our new definition of approximate lifting is inspired by a version of approximate liftings involving two witness distributions, proposed by Barthe and Olmedo [2013], Olmedo [2014].

{defi}

[Barthe and Olmedo [2013], Olmedo [2014]] Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$ be sub-distributions, $\varepsilon,\delta\in\mathbb{R}^{\geq 0}$ and $\mathrel{\mathcal{R}}$ be a binary relation over $A$ and $B$. An $(\varepsilon,\delta)$-approximate $2$-lifting of $\mu_{1}$ and $\mu_{2}$ for $\mathrel{\mathcal{R}}$ is a pair $(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}},\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})$ of sub-distributions over $A\times B$ s.t.

1. (1)

$\pi_{1}^{\sharp}(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})=\mu_{1}$ and $\pi_{2}^{\sharp}(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})=\mu_{2}$; 2. (2)

$\Delta_{\varepsilon}(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}},\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})\leq\delta$; and 3. (3)

$\operatorname{supp}(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})\subseteq\mathrel{\mathcal{R}}$ and $\operatorname{supp}(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})\subseteq\mathrel{\mathcal{R}}$.

We write $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(2)}_{\varepsilon,\delta}}\mu_{2}$ if there exists an $(\varepsilon,\delta)$-approximate ($2$-)lifting of $\mu_{1}$ and $\mu_{2}$ for $\mathrel{\mathcal{R}}$; the superscript $\cdot^{(2)}$ indicates that there are two witnesses $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ in this definition of lifting.

Combined with Lemma 1, a probabilistic computation $M:A\to{\mathbb{D}}(B)$ is $(\varepsilon,\delta)$-differentially private if and only if for every two adjacent inputs $a\mathrel{\phi}a^{\prime}$, there is an approximate lifting of the equality relation: $M(a)\mathrel{=^{(2)}_{\varepsilon,\delta}}M(a^{\prime})$.

$2$-liftings can be generalized by varying the notion of distance given by $\Delta_{\varepsilon}$; we will return to this point in Section 7. These liftings also satisfy useful theoretical properties, but some of the properties are not as general as we would like. For example, it is known that $2$-liftings satisfy the following mapping property.

Theorem 2** (Barthe et al. [2016a]).**

Let $\mu_{1}\in{\mathbb{D}}(A_{1})$, $\mu_{2}\in{\mathbb{D}}(A_{2})$ be distributions $f_{1}:A_{1}\to B_{1}$, $f_{2}:A_{2}\to B_{2}$ be surjective maps and $\mathrel{\mathcal{R}}$ be a binary relation on $B_{1}$ and $B_{2}$. Then

[TABLE]

where $a_{1}\mathrel{\mathrel{\mathcal{S}}}a_{2}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{\iff}}}f_{1}(a_{1})\mathrel{\mathcal{R}}f_{2}(a_{2})$.

This property can be used to pull back an approximate lifting on two distributions over $B_{1},B_{2}$ to an approximate lifting on two distributions over $A_{1},A_{2}$. For applications in program logics, $B_{1},B_{2}$ could be the domain of a program variable, $A_{1},A_{2}$ could be the set of memories, and $f_{1},f_{2}$ could project a memory to a program variable. While the mapping theorem is quite useful, it is puzzling why it only applies to surjective maps. For instance, this theorem cannot be used when the maps $f_{1},f_{2}$ inject a smaller space into a larger space.

For another example, there exist $2$-liftings of the following form, sometimes called the optimal subset coupling.

Theorem 3** (Barthe et al. [2016a]).**

Let $\mu\in{\mathbb{D}}(A)$ and consider two subsets $P_{1}\subseteq P_{2}\subseteq A$. Suppose that $P_{2}$ is a strict subset of $A$. Then, we have the following equivalence:

[TABLE]

where $a_{1}\mathrel{\mathrel{\mathcal{R}}}a_{2}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{\iff}}}a_{1}\in P_{1}\iff a_{2}\in P_{2}$.

In this construction, it is puzzling why the larger subset $P_{2}$ must be a strict subset of the domain $A$. For example, this theorem does not apply for $P_{2}=A$, but we may be able to construct the approximate lifting if we simply embed $A$ into a larger space $A^{\prime}$—even though $\mu$ has support over $A$! Furthermore, it is not clear why the subsets must be nested, nor is it clear why we can only relate $\mu$ to itself.

These shortcomings suggest that the definition of $2$-liftings may be problematic. While the distance condition appears to be the most constraining requirement, the marginal and support conditions are responsible for the main issues.

Witnesses must have support in the relation $\mathrel{\mathcal{R}}$.

For some relations $\mathrel{\mathcal{R}}$, there may be elements $a$ such that $a\mathrel{\mathrel{\mathcal{R}}}b$ does not hold for any $b$, or vice versa. It can be impossible to find witnesses with the correct marginals on these elements while satisfying the support condition, even if the distance condition is satisfied. At a high level, there are situations where it is possible to construct a pair $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ satisfying the distance requirement, but where $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ needs additional mass to achieve the marginal requirement for some element $b$. Adding this mass anywhere preserves the distance bound between $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$—since it only increases the mass of $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ while preserving the mass of $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$, and the distance bound is asymmetric—but if there is no element $a$ such that $a\mathrel{\mathrel{\mathcal{R}}}b$ then we cannot place this mass within the required support, and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ cannot be related by an approximate lifting.

No canonical choice of witnesses.

A related problem is that the marginal requirement only constrains one marginal of each witness distribution. Along the other component, the witnesses may place the mass anywhere on any pair in the relation. As a result, witnesses to an approximate lifting $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(2)}_{\varepsilon,\delta}}\mu_{2}$ are sometimes required to place mass outside of $\operatorname{supp}(\mu_{1})\times\operatorname{supp}(\mu_{2})$, even though intuitively only elements in the support of the related distributions should be relevant to the lifting. This theoretical flaw makes it difficult to establish basic mapping and support properties of approximate liftings.

{exa}

We illustrate these problems with a concrete example. Consider the geometric distribution with parameter $p=1/2$, a distribution over the natural numbers which models the distribution of number of flips of a fair coin before the coin first comes up tails. Formally, the distribution $\gamma\in{\mathbb{D}}(\mathbb{N})$ is defined by $\gamma(k)=1/2^{k+1}$. Consider the binary relation $\mathrel{\mathcal{R}}=\{(x_{1},x_{2})\mid x_{1}+1=x_{2}\}$ over $\mathbb{N}$. Now, $\gamma$ cannot be related to itself via an approximate lifting $\gamma\mathrel{\mathrel{\mathcal{R}}^{(2)}_{\varepsilon,\delta}}\gamma$ for any parameters $\varepsilon,\delta$. To see why, the second witness $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ must satisfy the second marginal condition at $k=0$, so it must put total weight $\gamma(0)=1/2$ on pairs of the form $(-,0)$. These pairs must belong to the relation $\mathrel{\mathcal{R}}$, but there is no $x_{1}\in\mathbb{N}$ such that $x_{1}+1=0$. However, there is an approximate lifting $\overline{\gamma}\mathrel{\overline{\mathrel{\mathcal{R}}}^{(2)}_{\ln(2),0}}\overline{\gamma}$, where $\gamma$ and $\mathrel{\mathcal{R}}_{+1}$ are extended to the integers $\mathbb{Z}$. For instance, the two joint distributions with support $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}(z,z+1)=1/2^{z+1}$ for $z\geq 0$, and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}(z,z+1)=1/2^{z+2}$ for $z\geq-1$ form witnesses.

This behavior is a sign that the notion of approximate lifting is not well-behaved: the support of $\overline{\gamma}$ remains the non-negative integers, but somehow embedding $\gamma$ into a larger space enables additional approximate liftings.

3. $\star$-Liftings and Strassen’s Theorem

To improve the theoretical properties of $2$-liftings, we propose a simple extension: allow witnesses to be distributions over a larger set.

{nota}

For a set $A$, we write $A^{\star}$ for $A\uplus\{\star\}$. For a distribution $\eta\in{\mathbb{D}}(C)$ and a subset $C^{\prime}\subseteq C$, we write $\eta_{|C^{\prime}}$ for the restriction of $\eta$ to $C^{\prime}$, i.e., the sub-distribution given by $\eta_{|C^{\prime}}(c)=\eta(c)$ for $c\in C^{\prime}$, and $\eta_{|C^{\prime}}(c)=0$ otherwise.

{defi}

[$\star$-lifting] Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$ be sub-distributions, $\varepsilon,\delta\in\mathbb{R}^{\geq 0}$ and $\mathrel{\mathcal{R}}$ be a binary relation over $A$ and $B$. An $(\varepsilon,\delta)$-approximate $\star$-lifting of $\mu_{1}$ and $\mu_{2}$ for $\mathrel{\mathcal{R}}$ is a pair of sub-distributions $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}\in{\mathbb{D}}(A\times B^{\star})$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}\in{\mathbb{D}}(A^{\star}\times B)$ s.t.

1. (1)

$\pi_{1}^{\sharp}(\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})=\mu_{1}$ and $\pi_{2}^{\sharp}(\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})=\mu_{2}$; 2. (2)

$\operatorname{supp}({\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}_{|A\times B}),\operatorname{supp}({\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}_{|A\times B})\subseteq\mathcal{R}$; and 3. (3)

$\Delta_{\varepsilon}(\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}},\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}})\leq\delta$, where $\overline{\eta_{\bullet}}$ is the extension of $\eta_{\bullet}$ to ${\mathbb{D}}(A^{\star}\times B^{\star})$ given by the evident inclusions from $A\times B^{\star}$ and $A^{\star}\times B$ to $A^{\star}\times B^{\star}$.

We write $\mu_{1}\mathrel{R^{(\star)}_{\varepsilon,\delta}}\mu_{2}$ if there exists an $(\varepsilon,\delta)$-approximate $\star$-lifting of $\mu_{1}$ and $\mu_{2}$ for $\mathrel{\mathcal{R}}$.

By adding an element $\star$, we address both problems discussed at the end of the previous section. First, for every $a\in A$ witnesses may place mass at $(a,\star)$, and for every $b\in B$ witnesses may place mass at $(\star,b)$. Second, $\star$ serves as a generic element where all mass outside the supports $\operatorname{supp}(\mu_{1})\times\operatorname{supp}(\mu_{2})$ may located, giving more control over the form of the witnesses. Formally, $\star$-liftings satisfy the following natural support property.

Lemma 4**.**

Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$ be distributions such that $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$ . Then, there are witnesses with support contained in $\operatorname{supp}(\mu_{1})^{\star}\times\operatorname{supp}(\mu_{2})^{\star}$.

Proof 3.1**.**

See Appendix, p. 7.5

+=

Proof 3.2** (Proof of ).**

Let $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ be any pair of witnesses to the approximate lifting. We will construct witnesses $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}},\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ with the desired support. For ease of notation, let $S_{i}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\operatorname{supp}(\mu_{i})$ for $i\in\{1,2\}$. Define:

[TABLE]

Evidently, $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ have support in $S_{1}^{\star}\times S_{2}^{\star}$. Additionally, it is straightforward to check that $\pi_{1}^{\sharp}(\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})=\pi_{1}^{\sharp}(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})=\mu_{1}$ and $\pi_{2}^{\sharp}(\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})=\pi_{2}^{\sharp}(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})=\mu_{2}$ so $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ have the desired marginals.

It only remains to check the distance condition. By the definition of the distance $\Delta_{\varepsilon}$, we know that there are non-negative values $\delta(a,b)$ such that (i) $\overline{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}(a,b)\leq e^{\varepsilon}\overline{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}(a,b)+\delta(a,b)$ and (ii) $\sum_{a,b}\delta(a,b)\leq\delta$. We can define new constants:

[TABLE]

Since $\overline{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}(\star,b)=\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}(\star,b)=0$ for all $b\in B^{\star}$, and $\overline{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}(a,b)=\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}(a,b)=0$ for all $b\notin S_{2}$, point (i) holds for the witnesses $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}},\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ and constants $\zeta(a,b)$. Since $\sum_{a,b}\zeta(a,b)=\sum_{a,b}\delta(a,b)\leq\delta$, point (ii) holds as well. Hence, $\Delta_{\varepsilon}(\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}},\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}})\leq\delta$ and we have witnesses for the desired approximate lifting.

3.1. Basic Properties

$\star$-liftings satisfy key properties enjoyed by existing notions of approximate lifting. To start, $\star$-liftings characterize differential privacy.111The proofs of the next two lemmas use an equivalence that we will soon prove in Theorem 9. This is purely for convenience—these proofs could also be performed separately, and in any case Theorem 9 does not use these lemmas so there is no circularity.

Lemma 5**.**

A randomized algorithm $P:A\to{\mathbb{D}}(B)$ is $(\varepsilon,\delta)$-differentially private w.r.t. ${\rm Phys.~{}Rev.~{}E}$ if for all $(a_{1},a_{2})\in{\rm Phys.~{}Rev.~{}E}$ we have $P(a_{1})\mathrel{=^{(\star)}_{\varepsilon,\delta}}P(a_{2})$. (Here, we are turning the equality relation on $B$ to an approximate lifting relating distributions over $B$.)

Proof 3.3**.**

See Appendix, p. 7.5

+=

Proof 3.4** (Proof of ).**

**($\Longrightarrow$): **

Let $P$ be $(\varepsilon,\delta)$-differentially private w.r.t. ${\rm Phys.~{}Rev.~{}E}$ and let $(a_{1},a_{2})\in{\rm Phys.~{}Rev.~{}E}$. Let $X$ be a subset of $B$. By definition of differential privacy, we have $P(a_{1})[X]\leq e^{\varepsilon}\cdot P(a_{2})[X]+\delta=e^{\varepsilon}\cdot P(a_{2})[(=)(X)]+\delta$. Recall that the image of a set $X$ under a binary relation is simply the set of all elements related to some element in $X$. In particular, $(=)(X)$ is just $X$. Hence, by application of Theorem 9, we have $P(a_{1})\mathrel{=^{(\star)}_{\varepsilon,\delta}}P(a_{2})$.

**($\Longleftarrow$): **

By application of Theorem 9, we have that

[TABLE]

This is the definition of $P$ being $(\varepsilon,\delta)$-differentially private w.r.t. ${\rm Phys.~{}Rev.~{}E}$.

The next lemma establishes several other basic properties of $\star$-liftings: monotonicity, and closure under relational and sequential composition.

Lemma 6**.**

• •

Let $\mu_{1}\in{\mathbb{D}}(A)$, $\mu_{2}\in{\mathbb{D}}(B)$, and $\mathrel{\mathcal{R}}$ be a binary relation over $A$ and $B$. If $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$, then for any $\varepsilon^{\prime}\geq\varepsilon$, $\delta^{\prime}\geq\delta$ and $\mathcal{S}\supseteq\mathcal{R}$, we have $\mu_{1}\mathrel{\mathrel{\mathcal{S}}^{(\star)}_{\varepsilon^{\prime},\delta^{\prime}}}\mu_{2}$.

• •

Let $\mu_{1}\in{\mathbb{D}}(A)$, $\mu_{2}\in{\mathbb{D}}(B)$, $\mu_{3}\in{\mathbb{D}}(C)$ and $\mathcal{R}$ (resp. $\mathcal{S}$) be a binary relation over $A$ and $B$ (resp. over $B$ and $C$). If $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$ and $\mu_{2}\mathrel{\mathrel{\mathcal{S}}^{(\star)}_{\varepsilon^{\prime},\delta^{\prime}}}\mu_{3}$, then $\mu_{1}\mathrel{(\mathrel{\mathcal{S}}\circ\mathrel{\mathcal{R}})^{(\star)}_{\varepsilon+\varepsilon^{\prime},\delta+e^{\varepsilon}\cdot\delta^{\prime}}}\mu_{3}$.

• •

For $i\in\{1,2\}$, let $\mu_{i}\in{\mathbb{D}}(A_{i})$ and $\eta_{i}:A_{i}\to{\mathbb{D}}(B_{i})$. Let $\mathrel{\mathcal{R}}$ (resp. $\mathrel{\mathcal{S}}$) be a binary relation over $A_{1}$ and $A_{2}$ (resp. over $B_{1}$ and $B_{2}$). If $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$ for some $\varepsilon,\delta\geq 0$ and for any $(a_{1},a_{2})\mathrel{\in}\mathrel{\mathcal{R}}$, we have $\eta_{1}(a_{1})\mathrel{\mathrel{\mathcal{S}}^{(\star)}_{\varepsilon^{\prime},\delta^{\prime}}}\eta_{2}(a_{2})$ for some $\varepsilon^{\prime},\delta^{\prime}\geq 0$, then

[TABLE]

Proof 3.5**.**

See Appendix, p. 7.5

+=

Proof 3.6** (Proof of ).**

• •

Immediate.

• •

Let $\overline{\varepsilon}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\varepsilon+\varepsilon^{\prime}$ and $\overline{\delta}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\delta+e^{\varepsilon}\cdot\delta^{\prime}$. By Theorem 9, it is sufficient to show that $\mu_{1}(X)\leq e^{\overline{\varepsilon}}\cdot\mu_{1}(\mathcal{S}(\mathcal{R}(X)))+\overline{\delta}$ for any set $X$. We have:

[TABLE]

• •

We know that $\exists\ \langle{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}},\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}\rangle\mathrel{\blacktriangleleft_{\varepsilon,\delta}^{\mathrel{\mathcal{R}}}}\langle{\mu_{1}}\mathrel{\&}{\mu_{2}}\rangle$. Likewise, for $a\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}(a_{1},a_{2})\in\mathcal{R}$, $\exists\ \langle{\eta_{{\scriptscriptstyle\vartriangleleft},a},\eta_{{\scriptscriptstyle\vartriangleright},a}}\rangle\mathrel{\blacktriangleleft_{\varepsilon^{\prime},\delta^{\prime}}^{\mathrel{\mathcal{S}}}}\langle{\eta_{1}(a_{1})}\mathrel{\&}{\eta_{2}(a_{2})}\rangle$. Let $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ be the following distribution constructors:

[TABLE]

and let $\xi_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\mathbb{E}_{{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}}[{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}]$ (resp. $\xi_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\mathbb{E}_{{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}}[{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}]$). We now prove that:

[TABLE]

The marginal and support conditions are immediate. The distance condition is obtained by an immediate application of the previous point.

3.2. Equivalence with Sato’s Definition

In recent work on verifying differential privacy over continuous distributions, Sato [2016] proposes an alternative definition of approximate lifting. In the special case of discrete distributions—where all events are measurable—his definition can be stated as follows.

{defi}

[Sato [2016]] Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$, $\mathrel{\mathcal{R}}$ be a binary relation over $A$ and $B$ and $\varepsilon,\delta\geq 0$. The distributions $\mu_{1}$ and $\mu_{2}$ are related by a (witness-free) $(\varepsilon,\delta)$-approximate lifting for $\mathrel{\mathcal{R}}$ if

[TABLE]

We write $\mathrel{\mathcal{R}}(X)=\{b\in B\mid\exists a\in X.\,(a,b)\in\mathrel{\mathcal{R}}\}\subseteq B$.

Notice that this definition has no witness distributions at all; instead, it uses a universal quantifier over all subsets. We can show that $\star$-liftings are equivalent to Sato’s definition in the case of discrete distributions. This equivalence is reminiscent of Strassen’s theorem from probability theory, which characterizes the existence of probabilistic couplings.

Theorem 7** (Strassen [1965]).**

Let $\mu_{1}\in{\mathbb{D}}(A)$, $\mu_{2}\in{\mathbb{D}}(B)$ be two proper distributions, and let $\mathcal{R}$ be a binary relation over $A$ and $B$. Then there exists a joint distribution $\mu\in{\mathbb{D}}(A\times B)$ with support in $\mathcal{R}$ such that $\pi_{1}^{\sharp}(\mu)=\mu_{1}$ and $\pi_{2}^{\sharp}(\mu)=\mu_{2}$ if and only if

[TABLE]

Our result (Theorem 9) can be viewed as a generalization of Strassen’s theorem to approximate couplings. The key ingredient in our proof is the max-flow min-cut theorem; we begin by reviewing the basic setting.

{defi}

[Flow network] A flow network is a structure $((V,E),\mathop{\top},\mathop{\perp},c)$ s.t. $\mathcal{N}=(V,E)$ is a loop-free directed graph without infinite simple path (or rays), $\mathop{\top}$ and $\mathop{\perp}$ are two distinct distinguished vertices of $\mathcal{N}$ s.t. no edge starts from $\mathop{\perp}$ and ends at $\mathop{\top}$, and $c:E\to\mathbb{R}^{+}\cup\{+\infty\}$ is a function assigning to each edge of $\mathcal{N}$ a capacity. The capacity $c$ is extended to $V^{2}$ by assigning capacity [math] to any pair $(u,v)$ s.t. $(u,v)\notin E$.

{defi}

[Flow] Given a flow network $\mathcal{N}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}((V,E),\mathop{\top},\mathop{\perp},c)$, a function $f:V^{2}\to\mathbb{R}$ is a flow for $\mathcal{N}$ iff

1. (1)

$\forall u,v\in V.\,f(u,v)\leq c(u,v)$, 2. (2)

$\forall u,v\in V.\,f(u,v)=-f(v,u)$, and 3. (3)

$\forall u\in V.\,u\notin\{\mathop{\top},\mathop{\perp}\}\implies\sum_{v\in V}f(u,v)=0$ (Kirchhoff’s Law).

The mass $|f|$ of a flow $f$ is defined as $|f|\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\sum_{v\in V}f(\mathop{\top},v)\in\mathbb{R}\{\cup+\infty\}$.

{defi}

[Cut] Given a flow network $\mathcal{N}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}((V,E),\mathop{\top},\mathop{\perp},c)$, a cut for $\mathcal{N}$ is any set $C\subseteq V$ that partition $V$ s.t. $\mathop{\top}\in V$ but $\mathop{\perp}\notin V$. The cut-set $\mathcal{E}(C)\subseteq E$ of a cut $C$ is the set of edges crossing the cut: $\{(u,v)\in E\mid u\in C,v\notin C\}$. The capacity $|C|\in\mathbb{R}^{\geq 0}\cup\{\infty\}$ of a cut $C$ is defined as $|C|\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\sum_{(u,v)\in\mathcal{E}(C)}c(u,v)$.

For finite flow networks, the maximum flow is equal to the minimum cut (see, e.g., Kleinberg and Tardos [2005]). Aharoni et al. [2011] generalize this theorem to networks with countable vertices and edges under certain conditions. We will use a consequence of their result.

Theorem 8** (Weak Countable Max-Flow Min-Cut).**

Let $\mathcal{N}$ be flow network with (a) no infinite directed paths and (b) finite total capacity leaving $\top$ and entering $\bot$. Then,

[TABLE]

and both supremum and infimum are achieved by some flow and cut, respectively.

We are now ready to prove an approximate version of Strassen’s theorem, thereby showing equivalence between $\star$-liftings and Sato’s liftings.

Theorem 9**.**

Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$, $\mathrel{\mathcal{R}}$ be a binary relation over $A$ and $B$ and $\varepsilon,\delta\in\mathbb{R}^{\geq 0}$. Then, $\mu_{1}\mathrel{R^{(\star)}_{\varepsilon,\delta}}\mu_{2}$ iff $\forall X\subseteq A.\,\mu_{1}(X)\leq e^{\varepsilon}\cdot\mu_{2}(\mathcal{R}(X))+\delta$.

Proof 3.7**.**

We detail the reverse direction; the forward direction is immediate. We can assume that $A$ and $B$ are countable; in the case where $A$ and $B$ are not both countable, we first consider the restriction of $\mu_{1}$ and $\mu_{2}$ to their respective supports—which are countable sets—and construct witnesses to the $\star$-lifting. The witnesses can then be extended to an approximate coupling of $\mu_{1}$ and $\mu_{2}$ by adding a null mass to the extra points.

Let $\omega\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}|{\mu_{2}}|+e^{-\varepsilon}\cdot\delta$ and let $\mathop{\top}$ and $\mathop{\perp}$ be fresh symbols. For any set $X$, define $X^{\mathop{\top}}$ and $X^{\mathop{\perp}}$ resp. as $\{x^{\mathop{\top}}\mid x\in X\}$ and $\{x^{\mathop{\perp}}\mid x\in X\}$. Let $\mathcal{N}$ be the flow network of Figure 1 whose resp. source and sink are $\mathop{\top}$ and $\mathop{\perp}$, whose set of vertices $V$ is $\{\mathop{\top},\mathop{\perp}\}\uplus(A^{\star})^{\mathop{\top}}\uplus(B^{\star})^{\mathop{\perp}}$, and whose set of edges $E$ is $E_{\mathop{\top}}\uplus E_{\mathop{\perp}}\uplus E_{\mathcal{R}}\uplus E_{\star}$ with

[TABLE]

Let $C$ be a cut of $\mathcal{N}$. In the following, we sometimes use $C$ to denote both the cut $C$ and its cut-set $\mathcal{E}(C)$. We check $|{C}|\geq\omega$. If $C\cap E_{\mathcal{R}}\neq\emptyset$ then $|{C}|=\infty$. Note that $C\cap E_{\star}=\emptyset$ implies $C\cap E_{\mathcal{R}}\neq\emptyset$. If $(\mathop{\top},\star^{\mathop{\top}})\in C$ and $(\mathop{\perp},\star^{\mathop{\perp}})\notin C$ then we must have $E_{\mathop{\top}}\subseteq C$. This implies that $|{C}|\geq\omega$ since $E_{\mathop{\top}}\uplus\{(\mathop{\top},\star^{\mathop{\top}})\}$ is a cut with capacity $\omega$. If $(\mathop{\top},\star^{\mathop{\top}})\notin C$ and $(\mathop{\perp},\star^{\mathop{\perp}})\in C$ then we have $|{C}|\geq\omega$ in the similar way as above. Otherwise (i.e. $C\cap E_{\mathcal{R}}=\emptyset$ and $E_{\star}\subseteq C$), for $C$ to be a cut, we must have $\mathcal{R}(A\mathrel{-}A^{\dagger})\subseteq B^{\dagger}$ where $A^{\dagger}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\{x\in A\mid(\mathop{\top},x^{\mathop{\top}})\in C\}$ and $B^{\dagger}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\{y\in B\mid(y^{\mathop{\perp}},\mathop{\perp})\in C\}$. Thus,

[TABLE]

Hence, $E_{\mathop{\top}}\uplus\{(\star^{\mathop{\perp}},\mathop{\perp})\}$ is a minimum cut with capacity $\omega$. By Theorem 8, we obtain a maximum flow $f$ with mass $\omega$. Note that the flow $f$ saturates the capacity of all edges in $E_{\mathop{\top}}$, $E_{\mathop{\perp}}$, and $E_{\star}$. Let $\hat{f}:(a,b)\in A^{\star}\times B^{\star}\mapsto f(a^{\mathop{\top}},b^{\mathop{\perp}})$. We now define the following distributions:

[TABLE]

We clearly have $\pi_{1}^{\sharp}(\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})=\mu_{1}$ and $\pi_{2}^{\sharp}(\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})=\mu_{2}$. Moreover, by construction of the flow network $\mathcal{N}$, $\operatorname{supp}(\hat{f}_{|A\times B})\subseteq\mathcal{R}$. Hence, $\operatorname{supp}({\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}_{|A\times B}),\operatorname{supp}({\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}_{|A\times B})\subseteq\mathcal{R}$. It remains to show that $\Delta_{\varepsilon}(\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}},\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}})\leq\delta$. Let $X$ be a subset of $A^{\star}\times B^{\star}$. Let $\overline{X_{a}}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\{a\in A\mid(a,\star)\in X\}$, $\overline{X_{b}}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\{b\in B\mid(\star,b)\in X\}$ and $\overline{X}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}X\cap(A\times B)$. Then,

[TABLE]

The last equality holds by Kirchhoff’s law: $\hat{f}[A\times\{\star\}]=\sum_{a\in A}f({a^{\mathop{\top}}},{\star^{\mathop{\perp}}})=f({\star^{\mathop{\perp}}},\mathop{\perp})=e^{-\varepsilon}\cdot\delta$.

+=

Proof 3.8** (Proof of ).**

To show the forward direction of Theorem 9, let $X\subseteq A$ and $\overline{\mathcal{R}(X)}=B\mathrel{-}\mathcal{R}(X)$. Then, we have

[TABLE]

as desired.

3.3. Alternative Proof of Approximate Strassen’s Theorem

We can provide an alternative, arguably simpler proof of the reverse direction of the approximate Strassen’s theorem (Theorem 9). Instead of relying on the max-flow min-cut theorem for countable networks by Aharoni et al. [2011], we apply the more standard result on finite networks and then pass from approximate liftings on finite restrictions of the two target distributions to an approximate lifting of the limit distributions, via a limiting argument. The results of this section have been formalized in the Coq proof assistant.222https://github.com/strub/xhl

We first start with a simple technical lemma.

Lemma 10**.**

Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$ (with respective support $S_{1}$ and $S_{2}$), $\mathrel{\mathcal{R}}$ be a binary relation over $A$ and $B$ and $\varepsilon,\delta\in\mathbb{R}^{\geq 0}$. Then, ${(\mu_{1})}_{|{S_{1}}}\mathrel{R^{(\star)}_{\varepsilon,\delta}}{(\mu_{2})}_{|{S_{2}}}$ implies $\mu_{1}\mathrel{R^{(\star)}_{\varepsilon,\delta}}\mu_{2}$.

Proof 3.9**.**

See Appendix, p. 7.5

+=

Proof 3.10** (Proof of ).**

If $\eta_{L}\in{\mathbb{D}}(S_{1}\times S_{2}^{\star})$, $\eta_{R}\in{\mathbb{D}}(S_{1}^{\star}\times S_{2})$ are the witnesses of ${(\mu_{1})}_{|{S_{1}}}\mathrel{R^{(\star)}_{\varepsilon,\delta}}{(\mu_{2})}_{|{S_{2}}}$, their extensions to ${\mathbb{D}}(A\times B^{\star})$ and ${\mathbb{D}}(A^{\star}\times B)$, padding with [math], form witnesses for $\mu_{1}\mathrel{R^{(\star)}_{\varepsilon,\delta}}\mu_{2}$.

We can now prove the theorem for distributions over finite domains:

Lemma 11** (Finite approximate Strassen’s theorem).**

Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$ be sub-distributions with finite supports, $\mathrel{\mathcal{R}}$ be a binary relation over $A$ and $B$ and $\varepsilon,\delta\in\mathbb{R}^{\geq 0}$. If for all $X\subseteq A$ we have $\mu_{1}[X]\leq e^{\varepsilon}\cdot\mu_{2}[\mathcal{R}(X)]+\delta$, then $\mu_{1}\mathrel{R^{(\star)}_{\varepsilon,\delta}}\mu_{2}$.

Proof 3.11**.**

See Appendix, p. 7.5

+=

Proof 3.12** (Proof of ).**

The case for distributions with finite domains can be proven using the same proof of Theorem 9, using the standard (finite) max-flow min-cut theorem. The result extends to distributions with finite supports via Lemma 10.

To extend the finite case to distributions over countable sets, we need a lemma that will allow us to assume that the witnesses are within a multiplicative factor of each other, except possibly on pairs with $\star$.

Lemma 12**.**

Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$ s.t. $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$. Then, there exists $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ witnessing the lifting s.t. for $(a,b)\in A\times B$, we have:

[TABLE]

Proof 3.13**.**

See Appendix, p. 7.5

+=

Proof 3.14** (Proof of ).**

Let $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$, $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ be witnesses of $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$. For $(a,b)\in A\times B$, let

[TABLE]

and define $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$, $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ as

[TABLE]

Note that $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ are well-defined as sub-distributions by the marginal conditions for $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$. To show that the witnesses are non-negative, let $a\in A$ and $b\in B$. We have:

[TABLE]

To show that the witnesses sum to at most $1$, we have

[TABLE]

Moreover, these witness distributions satisfy the claimed distance condition. Indeed for $(a,b)\in A\times B$, we have:

[TABLE]

It remains to prove that $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$, $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ are witnesses for $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$. The marginals conditions are obvious. For the support condition, let $a\in A$ and $b\in B$. Then, $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}(a,b)>0$ (resp. $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}(a,b)>0$) implies $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}(a,b)>0$ (resp. $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}(a,b)>0$) and hence that $a\mathrel{\mathcal{R}}b$.

The distance condition follows by a calculation. For $X\subseteq A^{\star}\times B^{\star}$, we have:

[TABLE]

To conclude, it suffices to show that $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}[X\cap(A\times\{\star\})]\leq\delta$. For that, let

[TABLE]

Then, we can bound

[TABLE]

Now, let $S\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\{(a,b)\in A^{\star}\times B^{\star}\mid e^{\varepsilon}\cdot\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}(a,b)<\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}(a,b)\}$. By the distance condition on $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$, we have $\overline{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}[S]-e^{\varepsilon}\cdot\overline{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}[S]\leq\delta$. Hence we conclude the desired distance condition:

[TABLE]

We can now prove the reverse direction of Theorem 9.

Proof 3.15** (Alternative proof of Theorem 9).**

By Lemma 10, without loss of generality, we can assume that $A$ and $B$ are countable. Hence, there exists a family $\{A_{n}\}_{n}$ (resp. $\{B_{n}\}_{n}$) of increasing finite subsets of $A$ s.t. $\cup_{i}A_{i}=A$ (resp. of $B$ s.t. $\cup_{i}B_{i}=B$). For $n\in\mathbb{N}$, we denote by $\mu_{1}^{n}$ and $\mu_{2}^{n}$ the domain restrictions of $\mu_{1}$ to $A_{n}$ and $\mu_{2}$ to $B_{n}$, i.e., $\mu_{1}^{n}(a)=\mu_{1}(a)$ if $a\in A_{n}$, [math] otherwise and $\mu_{2}^{n}(b)=\mu_{2}(b)$ if $b\in B_{n}$, [math] otherwise.

Fix $n\in\mathbb{N}$ and let $X\subseteq A$. We have:

[TABLE]

Hence, by Lemma 11, we have $\mu_{1}^{n}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta_{n}}}\mu_{2}^{n}$. By Lemma 12, we can moreover assume that $\mu_{1}^{n}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta_{n}}}\mu_{2}^{n}$ is witnessed by sub-distributions $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}^{n}$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}^{n}$ such that

[TABLE]

Since $\mathrel{\mathcal{R}}(X)\cap\overline{B_{n}}\xrightarrow[n\to\infty]{}\emptyset$, we also have $\delta_{n}\xrightarrow[n\to\infty]{}\delta$.

As a countable product of a sequentially compact sets, $[0,1]^{A\times B^{\star}}$ and $[0,1]^{A^{\star}\times B}$ are sequentially compact, and we can find a subsequence of indices $\{\omega_{n}\}_{n\in\mathbb{N}}$ s.t. both $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}^{\omega_{n}}$, $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}^{\omega_{n}}$ resp. converge pointwise to sub-distributions $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$.

We now prove that these sub-distributions are witnesses for $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$. It is clear that the supports of ${\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}_{|{A\times B}}$ and ${\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}_{|{A\times B}}$ are included in $\mathrel{\mathcal{R}}$. We now detail the marginals and distance conditions. First, note that

[TABLE]

Hence, Equation (2) yields

[TABLE]

Now, for the first marginal, let $a\in A$. We have:

[TABLE]

From (3a), it is clear that $b\in B^{\star}\mapsto\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}^{\omega_{n}}(a,b)$ is absolutely dominated by the summable function [b\in B^{\star}\mapsto\text{e^{\varepsilon}\cdot\mu_{2}(b)$if$b\in B$, else$1}]. By the dominated convergence theorem, we can swap the limit and summation:

[TABLE]

For the second marginal, let $b\in B$. We have:

[TABLE]

From (3b), we have that $a\in A^{\star}\mapsto\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}^{\omega_{n}}(a,b)$ is absolutely dominated by the summable function: [a\in A^{\star}\mapsto\text{\mu_{1}(a)$if$a\in A$, else$1}]. Again by the dominated convergence theorem, we can swap the limit and summation:

[TABLE]

We are left to prove the distance condition, i.e., that for any $X\subseteq A^{*}\times B^{*}$, we have $\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}[X]-e^{\varepsilon}\cdot\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}[X]\leq\delta$. First, note that $\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}[X]=\lim_{n\to\infty}\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}^{\omega_{n}}[X]$ and $\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}[X]=\lim_{n\to\infty}\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}^{\omega_{n}}[X]$. Indeed, for $\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}[X]$, we have

[TABLE]

where $X_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}^{a}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\{b\in B^{\star}\mid(a,b)\in X\}$. The first application of the dominated convergence theorem uses, like for the first marginal condition, the function [b\in B^{\star}\mapsto\text{e^{\varepsilon}\cdot\mu_{2}(b)$if$b\in B$, else$1}] as the dominating function (using Equation (3a)). The second application of the dominated convergence theorem uses [a\in A^{\star}\mapsto\text{\mu_{1}(a)$if$a\in A$, else$0}] as the dominating function. Indeed, if $a\in A$, then

[TABLE]

Likewise, for $\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}[X]$, we have

[TABLE]

where $X_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}^{b}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\{a\in A^{\star}\mid(a,b)\in X\}$. Here, the first application of the dominated convergence theorem uses, as for the second marginal condition, the function [a\in A^{\star}\mapsto\text{\mu_{1}(a)$if$a\in A$, else$1}] as the dominating function (using equation (3b)). The second application of the dominated convergence theorem uses [b\in B^{\star}\mapsto\text{\mu_{2}(b)$if$b\in B$, else$0}] as the dominating function. Indeed, if $b\in B$, then

[TABLE]

Hence, we can conclude the distance condition by taking limits:

[TABLE]

This proof constructs witnesses to an approximate lifting relating two distributions $(\mu_{1},\mu_{2})$ given a sequence of approximate liftings relating pairs of finite restrictions of $\mu_{1}$ and $\mu_{2}$. Using essentially the same argument, we can construct an approximate lifting relating $(\mu_{1},\mu_{2})$ given a sequence of approximate liftings relating pairs of distributions converging to $\mu_{1}$ and $\mu_{2}$; the main difference is that a slightly more general form of the dominated convergence theorem is needed (see Hsu [2017, Lemma 5.1.7] for details).

4. Properties of $\star$-Liftings

Our main theorem can be used to show several natural properties of $\star$-liftings. To begin, we can improve the mapping property from Theorem 2, lifting the requirement that the maps must be surjective.

Lemma 13**.**

Let $\mu_{1}\in{\mathbb{D}}(A_{1})$, $\mu_{2}\in{\mathbb{D}}(A_{2})$, $f_{1}:A_{1}\to B_{1}$, $f_{2}:A_{2}\to B_{2}$ and $\mathrel{\mathcal{R}}$ a binary relation on $B_{1}$ and $B_{2}$. Let $\mathrel{\mathcal{S}}$ such that $a_{1}\mathrel{\mathrel{\mathcal{S}}}a_{2}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{\iff}}}f_{1}(a_{1})\mathrel{\mathcal{R}}f_{2}(a_{2})$. Then

[TABLE]

Proof 4.1**.**

See Appendix, p. 7.5

+=

Proof 4.2** (Proof of ).**

**($\Longrightarrow$): **

Assume that $f_{1}^{\sharp}(\mu_{1})\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}f_{2}^{\sharp}(\mu_{2})$ and let $X\subseteq A_{1}$. Then,

[TABLE]

Hence, by Theorem 9, $\mu_{1}\mathrel{\mathrel{\mathcal{S}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$.

**($\Longleftarrow$): **

Assume that $\mu_{1}\mathrel{\mathrel{\mathcal{S}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$ and let $X\subseteq A_{2}$. Then,

[TABLE]

Hence, by Theorem 9, $f_{1}^{\sharp}(\mu_{1})\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}f_{2}^{\sharp}(\mu_{2})$.

Similarly, we can generalize the existing rules for up-to-bad reasoning (cf. Barthe et al. [2016a, Theorem 13]), which restrict the post-condition to be equality. There are two versions: the conditional event is either on the left side, or the right side. Note that the resulting indices $\overline{\delta}$ are different in the two cases. We write $\overline{\theta}$ for the complement of $\theta$.

Lemma 14**.**

Let $\mu_{1}\in{\mathbb{D}}(A)$, $\mu_{2}\in{\mathbb{D}}(B)$, $\theta\subseteq A$ and $\mathcal{R}\subseteq A\times B$. Assume that $\mu_{1}\mathrel{(\theta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}\implies\mathrel{\mathcal{R}})^{(\star)}_{\varepsilon,\delta}}\mu_{2}$ for some parameters $\varepsilon,\delta\geq 0$. Then $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\overline{\delta}}}\mu_{2}$, where $\overline{\delta}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\delta+\mu_{1}[\overline{\theta}]$.

Proof 4.3**.**

See Appendix, p. 7.5

+=

Proof 4.4** (Proof of ).**

By Theorem 9, it is sufficient to prove that

[TABLE]

for any $X\subseteq A$. By direct computation:

[TABLE]

Lemma 15**.**

Let $\mu_{1}\in{\mathbb{D}}(A)$, $\mu_{2}\in{\mathbb{D}}(B)$, $\theta\subseteq B$ and $\mathcal{R}\subseteq A\times B$. Assume that $\mu_{1}\mathrel{(\theta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}\implies\mathrel{\mathcal{R}})^{(\star)}_{\varepsilon,\delta}}\mu_{2}$ for some parameters $\varepsilon,\delta\geq 0$. Then, $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\overline{\delta}}}\mu_{2}$, where $\overline{\delta}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\delta+e^{\varepsilon}\cdot\mu_{2}[\overline{\theta}]$.

Proof 4.5**.**

See Appendix, p. 7.5

+=

Proof 4.6** (Proof of ).**

By Theorem 9, it is sufficient to prove that

[TABLE]

for any $X\subseteq A$. Let $X$ be such a set, then:

[TABLE]

As a consequence, an approximately lifted relation can be conjuncted with a one-sided predicate if the $\delta$ parameter is increased. This principle is useful for constructing approximate liftings based on accuracy bounds. For instance, suppose that we have an approximate lifting of $\mathcal{R}$ relating two distributions, $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$. If $\theta_{a}$ is an event that happens with high probability in the first distribution $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$—say, a certain noise variable is at most $100$—we can incorporate $\theta_{a,{\scriptscriptstyle\vartriangleleft}}$ into the approximate lifting by increasing the $\delta$ parameter by the probability that $\theta_{a}$ fails to hold in $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$. When reasoning in terms of approximate couplings, intuitively we can “assume” $\theta_{a}$ holds by “paying” with an increase in $\delta$. A similar property holds for the second distribution $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$.

We formalize these constructions with the following lemma.

Lemma 16**.**

Let $\mu_{1}\in{\mathbb{D}}(A)$, $\mu_{2}\in{\mathbb{D}}(B)$, $\theta_{a}\subseteq A$, $\theta_{b}\subseteq B$ and $\mathcal{R}\subseteq A\times B$. Assume that $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$. Then, $\mu_{1}\mathrel{(\theta_{a,{\scriptscriptstyle\vartriangleleft}}\cap\mathrel{\mathcal{R}})^{(\star)}_{\varepsilon,\delta_{a}}}\mu_{2}$ and $\mu_{1}\mathrel{(\theta_{b,{\scriptscriptstyle\vartriangleright}}\cap\mathrel{\mathcal{R}})^{(\star)}_{\varepsilon,\delta_{b}}}\mu_{2}$ where $\delta_{a}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\delta+\mu_{1}[\overline{\theta_{a}}]$ and $\delta_{b}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\delta+e^{\varepsilon}\cdot\mu_{2}[\overline{\theta_{b}}]$.

Proof 4.7**.**

See Appendix, p. 7.5

+=

Proof 4.8** (Proof of ).**

From $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$ and Lemma 6, we have $\mu_{1}\mathrel{\mathrel{\mathcal{S}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$ where $\mathcal{S}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\theta_{a,{\scriptscriptstyle\vartriangleleft}}\implies\theta_{a,{\scriptscriptstyle\vartriangleleft}}\cap\mathcal{R}$. Hence, by Lemma 14, we obtain $\mu_{1}\mathrel{(\theta_{a,{\scriptscriptstyle\vartriangleleft}}\cap\mathcal{R})^{(\star)}_{\varepsilon,\delta_{a}}}\mu_{2}$. Using similar reasoning with $\theta_{b,{\scriptscriptstyle\vartriangleright}}\implies\theta_{b,{\scriptscriptstyle\vartriangleright}}\cap\mathcal{R}$ and Lemma 15, we have $\mu_{1}\mathrel{(\theta_{b,{\scriptscriptstyle\vartriangleright}}\cap\mathcal{R})^{(\star)}_{\varepsilon,\delta_{b}}}\mu_{2}$.

$\star$-liftings also support a significant generalization of optimal subset coupling. Unlike the known construction for $2$-liftings (Theorem 3), the two subsets need not be nested, and either subset may be the entire domain. Furthermore, the distributions $\mu_{1},\mu_{2}$ need not be the same, or even have the same domain. Finally, the equivalence is valid for any parameters $(\varepsilon,\delta)$, not just $\delta=0$.

Theorem 17**.**

Let $\mu_{1}\in{\mathbb{D}}(A_{1})$, $\mu_{2}\in{\mathbb{D}}(A_{2})$ and consider two subsets $P_{1}\subseteq A_{1},P_{2}\subseteq A_{2}$. Then, we have the following equivalence:

[TABLE]

where $a_{1}\mathrel{\mathrel{\mathcal{R}}}a_{2}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{\iff}}}a_{1}\in P_{1}\iff a_{2}\in P_{2}$.

Proof 4.9**.**

Immediate by Theorem 9.

We can then recover the existing notion of optimal subset coupling [Barthe et al., 2016a] for $\star$-liftings as a special case.

Corollary 18** (Barthe et al. [2016a]).**

Let $\mu\in{\mathbb{D}}(A)$ and consider two nested subsets $P_{2}\subseteq P_{1}\subseteq A$. Then, we have the following equivalence:

[TABLE]

where $a_{1}\mathrel{\mathrel{\mathcal{R}}}a_{2}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{\iff}}}a_{1}\in P_{1}\iff a_{2}\in P_{2}$.

Proof 4.10**.**

Immediate by Theorem 17, noting that

[TABLE]

is automatic since $P_{2}\subseteq P_{1}$ implies $\mathbb{P}_{\mu}[{A\mathrel{-}P_{1}}]\leq\mathbb{P}_{\mu}[{A\mathrel{-}P_{2}}]$. Note that there is no longer a need for $P_{1}$ to be a strict subset of $A$.

Finally, we can directly extend known composition theorems from differential privacy to $\star$-liftings. This connection is quite useful for transferring existing composition results from the privacy literature to approximate liftings. We first define a general template describing how the privacy parameters $\varepsilon,\delta$ decay under sequential composition.

{defi}

Let $\mathbb{R}^{\geq 0}_{2}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\mathbb{R}^{\geq 0}\times\mathbb{R}^{\geq 0}$ and let $(\mathbb{R}^{\geq 0}_{2})^{*}$ be the set of finite sequences over pairs of non-negative reals. A map $r:(\mathbb{R}^{\geq 0}_{2})^{*}\to\mathbb{R}^{\geq 0}_{2}$ is a DP-composition rule if for all sets $A,D$, adjacency relations $\phi\subseteq D\times D$, and families of functions $\{f_{i}:D\times A\to{\mathbb{D}}(A)\}_{i<n}$, the following implication holds: if for every initial value $a\in A$ and $i<n$, $f_{i}(-,a):D\to{\mathbb{D}}(A)$ is $(\varepsilon_{i},\delta_{i})$-differentially private w.r.t. $\phi$, then $F(-,a)$ is $(\varepsilon^{*},\delta^{*})$-differentially private w.r.t. $\phi$ and any initial value $a\in A$ where $F:(d,a)\mapsto(\bigcirc_{i<n}\mathrel{(f_{i}(d,-))^{\sharp}})({{\mathbb{1}}_{a}})$ is the $n$-fold composition of the functions $[f_{i}]_{i<n}$ and $(\varepsilon^{*},\delta^{*})\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}r([(\varepsilon_{i},\delta_{i})]_{i<n})$.

Lemma 19**.**

Let $r:(\mathbb{R}^{\geq 0}_{2})^{*}\to\mathbb{R}^{\geq 0}_{2}$ be a DP-composition rule. Let $n\in\mathbb{N}$ and assume given two families of sets $\{A_{i}\}_{i\leq n}$ and $\{B_{i}\}_{i\leq n}$, together with a family of binary relations $\{\mathcal{R}(i)\subseteq A_{i}\times B_{i}\}_{i\leq n}$. Let $\{g_{i}:A_{i}\to{\mathbb{D}}(A_{i+1})\}_{i<n}$ and $\{h_{i}:B_{i}\to{\mathbb{D}}(B_{i+1})\}_{i<n}$ be two families of functions s.t. for all $i<n$ and $(a,b)\in\mathcal{R}(i)$, we have:

1. (1)

$g_{i}(a)\mathrel{\mathrel{\mathcal{R}(i+1)^{(\star)}_{\varepsilon_{i},\delta_{i}}}}h_{i}(b)$* for some parameters $\varepsilon_{i},\delta_{i}\geq 0$, and* 2. (2)

$g_{i}(a)$* and $h_{i}(b)$ are proper distributions.*

Then for $(a_{0},b_{0})\in\mathcal{R}(0)$, there exists a $\star$-lifting

[TABLE]

where $G:A_{0}\to{\mathbb{D}}(A_{n})$ and $H:B_{0}\to{\mathbb{D}}(B_{n})$ are the $n$-fold compositions of $[g_{i}]_{i\leq n}$ and $[h_{i}]_{i\leq n}$ respectively—i.e. $G(a)\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}(\bigcirc_{i<n}\mathrel{g_{i}^{\sharp}})({{\mathbb{1}}_{a}})$ and $H(b)\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}(\bigcirc_{i<n}\mathrel{h_{i}^{\sharp}})({{\mathbb{1}}_{b}})$—and $(\varepsilon^{*},\delta^{*})\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}r([(\varepsilon_{i},\delta_{i})]_{i<n})$.

Proof 4.11**.**

We assume that $A_{i}=A$ is the same for all $i$, and $B_{i}=B$ is the same for all $i$. This is without loss of generality, since when $A_{i}$ and $B_{i}$ vary with $i$ we may work with the disjoint unions $\sqcup_{i}A_{i}$ and $\sqcup_{i}B_{i}$ by restricting each $\mathcal{R}(i)$ to only relate pairs that are both in the $i$-th component. Define $D={\{0,1\}}=\{\mathit{tt},\mathit{ff}\}$ and $\phi=\{(\mathit{tt},\mathit{ff})\}\subseteq D\times D$.

For every $i<n$ and $(a_{i},b_{i})\in\mathcal{R}(i)$, the definition of $\star$-lifting gives two distributions $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}[a_{i},b_{i}],\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}[a_{i},b_{i}]$ witnessing $g_{i}(a)\mathrel{\mathrel{\mathcal{R}(i+1)^{(\star)}_{\varepsilon_{i},\delta_{i}}}}$. We regard both witness distributions as elements of ${\mathbb{D}}(A^{\star}\times B^{\star})$ via the evident embeddings. We define the maps $f_{i}:D\times(A^{\star}\times B^{\star})\to{\mathbb{D}}(A^{\star}\times B^{\star})$ by cases:

[TABLE]

where $\times$ denotes the product distribution and $(a_{i},b_{i})\in\mathcal{R}(i)$; otherwise, $f_{i}(-,(a,b))=0$.

Now for all $(a,b)\in A^{\star}\times B^{\star}$, the map $f_{i}(-,(a,b)):D\to{\mathbb{D}}(A^{\star}\times B^{\star})$ is $(\varepsilon_{i},\delta_{i})$-differentially private with respect to $\phi$ by the distance property on $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}[a_{i},b_{i}]$ and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}[a_{i},b_{i}]$ (and by definition when $(a_{i},b_{i})\notin\mathcal{R}(i)$). Hence, the DP-composition rule implies that $F:(d,(a,b))\mapsto(\bigcirc_{i<n}\mathrel{(f_{i}(d,-))^{\sharp}})({{\mathbb{1}}_{(a,b)}})$ is $(\varepsilon^{*},\delta^{*})$-differentially private with respect to $\phi$ for any $(a,b)\in A^{\star}\times B^{\star}$. For any $(a_{0},b_{0})\in\mathcal{R}(0)$, we claim that that $F(\mathit{tt},(a_{0},b_{0}))$ and $F(\mathit{ff},(a_{0},b_{0}))$ witness the desired approximate lifting

[TABLE]

The support and marginal conditions are not hard to show, and the distance condition follows from differential privacy of $F(-,(a_{0},b_{0})):D\to{\mathbb{D}}(A^{\star}\times B^{\star})$.

Some of the more sophisticated composition results from differential privacy—for instance, the advanced composition theorem by Dwork et al. [2010]—do not apply to arbitrary adjacency relations $\phi$, but only symmetric relations. Lemma 19 cannot lift such theorems to composition principles for approximate liftings. In Section 6 we will remedy this problem by working with a symmetric version of $\star$-lifting.

5. Comparison with Prior Approximate Liftings

Now that we have seen $\star$-liftings, we briefly consider other definitions of approximate liftings. We have already seen $2$-liftings, which involve two witnesses (Definition 1). Evidently, $\star$-liftings strictly generalize $2$-liftings.

Theorem 20**.**

For all binary relations $\mathrel{\mathcal{R}}$ over $A$ and $B$ and parameters $\varepsilon,\delta\geq 0$, we have $\mathrel{\mathrel{\mathcal{R}}^{(2)}_{\varepsilon,\delta}}\subseteq\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}.$ There exist relations and parameters where the inclusion is strict.

Proof 5.1**.**

The inclusion $\mathrel{\mathrel{\mathcal{R}}^{(2)}_{\varepsilon,\delta}}\subseteq\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}$ is immediate. We have a strict inclusion $\mathrel{\mathrel{\mathcal{R}}^{(2)}_{\varepsilon,\delta}}\subsetneq\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}$ even for $\delta=0$ by considering the optimal subset coupling from Theorem 3. Consider a distribution $\mu$ over set $A$, and let $P_{1}\subseteq P_{2}=A$. There is an $(\varepsilon,0)$-approximate $\star$-lifting (by Theorem 17), but a $(\varepsilon,0)$-approximate $2$-lifting does not exist if $\mu$ has non-zero mass outside of $P_{1}$: the first witness $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ must place non-zero mass at $(a_{1},a_{2})$ with $a_{1}\notin P_{1}$ in order to have $\pi_{1}^{\sharp}(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})=\mu$, but we must have $a_{2}\notin P_{2}$ for the support requirement, and there is no such $a_{2}$.

We can also compare $\star$-liftings with the original definitions of $(\varepsilon,\delta)$-approximate lifting by Barthe et al. [2013]. They introduce two notions, a symmetric lifting and an asymmetric lifting, each using a single witness distribution. We will focus on the asymmetric version here, and return to the symmetric version in Section 6.

{defi}

[Barthe et al. [2013]] Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$ be sub-distributions, $\varepsilon,\delta\in\mathbb{R}^{\geq 0}$ and $\mathrel{\mathcal{R}}$ be a binary relation over $A$ and $B$. An $(\varepsilon,\delta)$-approximate $1$-lifting of $\mu_{1}$ and $\mu_{2}$ for $\mathrel{\mathcal{R}}$ is a sub-distribution $\mu\in{\mathbb{D}}(A\times B)$ s.t.

1. (1)

$\pi_{1}^{\sharp}(\mu)\leq\mu_{1}$ and $\pi_{2}^{\sharp}(\mu)\leq\mu_{2}$; 2. (2)

$\Delta_{\varepsilon}(\mu_{1},\pi_{1}^{\sharp}(\mu))\leq\delta$; and 3. (3)

$\operatorname{supp}(\mu)\subseteq\mathrel{\mathcal{R}}$.

In the first point we take the point-wise order on sub-distributions: if $\mu$ and $\mu^{\prime}$ are sub-distributions over $X$, then $\mu\leq\mu^{\prime}$ when $\mu(x)\leq\mu^{\prime}(x)$ for all $x\in X$. We will write $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(1)}_{\varepsilon,\delta}}\mu_{2}$ if there exists an $(\varepsilon,\delta)$-approximate $1$-lifting of $\mu_{1}$ and $\mu_{2}$ for $\mathrel{\mathcal{R}}$; the superscript $\cdot^{(1)}$ indicates that there is one witness for this lifting.

$1$-liftings bear a close resemblance to probabilistic couplings from probability theory, which also have a single witness. However, $1$-liftings are more awkward to manipulate and less well-understood theoretically than their $2$-lifting cousins—basic properties such as mapping (Lemma 13) are not known to hold; the subset coupling (Theorem 3) is not known to exist. Somewhat surprisingly, $1$-liftings are equivalent to $\star$-liftings and hence by Theorem 9, also to Sato’s approximate lifting.

Theorem 21**.**

For all binary relations $\mathrel{\mathcal{R}}$ over $A$ and $B$ and parameters $\varepsilon,\delta\geq 0$, we have $\mathrel{\mathrel{\mathcal{R}}^{(1)}_{\varepsilon,\delta}}=\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}.$

Proof 5.2**.**

See Appendix, p. 7.5

+=

Proof 5.3** (Proof of ).**

Suppose that $(\mu_{L},\mu_{R})$ are witnesses to $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$. Define the witness $\eta\in{\mathbb{D}}(A\times B)$ as the point-wise minimum: $\eta(a,b)=\min(\mu_{L}(a,b),\mu_{R}(a,b))$. We will show that $\eta$ is a witness to $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(1)}_{\varepsilon,\delta}}\mu_{2}$.

The support condition follows from the support condition for $(\mu_{L},\mu_{R})$. The marginal conditions $\pi_{1}^{\sharp}(\eta)\leq\mu_{1}$ and $\pi_{2}^{\sharp}(\eta)\leq\mu_{2}$ also follow by the marginal conditions for $(\mu_{L},\mu_{R})$. The only thing to check is the distance condition. By the distance condition on $(\mu_{L},\mu_{R})$, there exist non-negative values $\delta(a,b)$ such that

[TABLE]

and $\sum_{a,b}\delta(a,b)\leq\delta$. So, $\mu_{R}(a,b)\geq\exp(-\varepsilon)(\mu_{L}(a,b)-\delta(a,b))$. Now let $S\subseteq A$ be any subset. Then:

[TABLE]

Thus, $\eta$ witnesses $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(1)}_{\varepsilon,\delta}}\mu_{2}$, so $\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\subseteq\mathrel{\mathrel{\mathcal{R}}^{(1)}_{\varepsilon,\delta}}$.

The other direction is more interesting. Let $\eta\in{\mathbb{D}}(A\times B)$ be the witness for $\mathrel{\mathrel{\mathcal{R}}^{(1)}_{\varepsilon,\delta}}$. By the distance condition $\Delta_{\varepsilon}(\mu_{1},\pi_{1}^{\sharp}\eta)\leq\delta$, there exist non-negative values $\delta(a)$ such that

[TABLE]

with equality when $\delta(a)$ is strictly positive, and $\sum_{a\in A}\delta(a)\leq\delta$. Define two witnesses $\mu_{L}\in{\mathbb{D}}(A\times B^{\star}),\mu_{R}\in{\mathbb{D}}(A^{\star}\times B)$ as follows:

[TABLE]

(As usual, if any denominator is zero, we take the probability to be zero as well.)

The support condition follows from the support condition of $\eta$. The marginal conditions hold by definition. Note that all probabilities are non-negative. For $\mu_{L}$, note that if $\delta(a)>0$ then $\mu_{1}(a)-\delta(a)=\exp(\varepsilon)\pi_{1}^{\sharp}\eta(a)\geq 0$ and hence

[TABLE]

assuming $\pi_{1}^{\sharp}\eta(a)>0$; if $\pi_{1}^{\sharp}\eta(a)=0$ then $\mu_{L}(a,\star)=0$. For $\mu_{R}$, non-negativity holds because $\pi_{2}^{\sharp}\eta\leq\mu_{2}$.

We show the distance bound. Note that when $a,b\neq\star$, by definition $\mu_{L}(a,b)$ and $\mu_{R}(a,b)$ are both strictly positive or both equal to zero, and $\eta(a,b)$ is strictly positive or equal to zero accordingly. If $\mu_{L}(a,b),\mu_{R}(a,b),\eta(a,b)$ are all strictly positive, then we know

[TABLE]

Thus we always have

[TABLE]

We can also bound the mass on points $(a,\star)$. Let $S\subseteq A$ be any subset. Then:

[TABLE]

So $\Delta_{\varepsilon}(\overline{\mu_{L}},\overline{\mu_{R}})\leq\delta$ as desired, and we have witnesses to $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$. Hence, $\mathrel{\mathrel{\mathcal{R}}^{(1)}_{\varepsilon,\delta}}\subseteq\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}$.

6. Symmetric $\star$-Lifting

The approximate liftings we have considered so far are all asymmetric. For instance, the approximate lifting $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$ may not imply the lifting $\mu_{2}\mathrel{(\mathrel{\mathcal{R}}^{-1})^{(\star)}_{\varepsilon,\delta}}\mu_{1}$. Given witnesses $(\mu_{L},\mu_{R})$ to the first lifting, we may consider the witnesses $(\nu_{L},\nu_{R})=(\mu_{R}^{\top},\mu_{L}^{\top})$ where the transpose map $(-)^{\top}:{\mathbb{D}}(A\times B)\to{\mathbb{D}}(B\times A)$ is defined in the obvious way. Then $(\nu_{L},\nu_{R})$ almost witness the second lifting—the marginal and support conditions holds, but the distance bound is in the wrong direction:

[TABLE]

In general, we cannot bound $\Delta_{\varepsilon}(\nu_{L},\nu_{R})$ and the symmetric lifting $\mu_{2}\mathrel{(\mathrel{\mathcal{R}}^{-1})^{(\star)}_{\varepsilon,\delta}}\mu_{1}$ may not hold. To recover symmetry, we can define a symmetric version of $\star$-lifting. {defi}[Symmetric $\star$-lifting] Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$ be sub-distributions, $\varepsilon,\delta\in\mathbb{R}^{\geq 0}$ and $\mathrel{\mathcal{R}}$ be a binary relation over $A$ and $B$. An $(\varepsilon,\delta)$-approximate symmetric $\star$-lifting of $\mu_{1}$ and $\mu_{2}$ for $\mathrel{\mathcal{R}}$ is a pair of sub-distributions $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}\in{\mathbb{D}}(A\times B^{\star})$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}\in{\mathbb{D}}(A^{\star}\times B)$ s.t.

1. (1)

$\pi_{1}^{\sharp}(\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})=\mu_{1}$ and $\pi_{2}^{\sharp}(\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})=\mu_{2}$; 2. (2)

$\operatorname{supp}({\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}_{|A\times B}),\operatorname{supp}({\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}_{|A\times B})\subseteq\mathcal{R}$; and 3. (3)

$\Delta_{\varepsilon}(\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}},\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}})\leq\delta,\Delta_{\varepsilon}(\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}},\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}})\leq\delta$, where $\overline{\eta_{\bullet}}$ is the canonical lifting of $\eta_{\bullet}$ to $A^{\star}\times B^{\star}$.

We write $\mu_{1}\mathrel{\overline{R}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$ if there exists an $(\varepsilon,\delta)$-approximate symmetric lifting of $\mu_{1}$ and $\mu_{2}$ for $\mathrel{\mathcal{R}}$.

Symmetric $\star$-lifting is a special case of $\star$-lifting that can capture differential privacy under when the adjacency relation $\phi$ is symmetric: a probabilistic computation $M:A\to{\mathbb{D}}(B)$ is $(\varepsilon,\delta)$-differentially private if and only if for every two adjacent inputs $a\mathrel{\phi}a^{\prime}$, there is an approximate lifting of the equality relation: $M(a)\mathrel{\overline{(=)}^{(\star)}_{\varepsilon,\delta}}M(a^{\prime})$. Unfortunately, the more advanced properties in Section 4 do not all hold when moving to symmetric liftings. However, we can show that symmetric $\star$-liftings are equivalent to the symmetric version of $1$-witness lifting proposed by Barthe et al. [2013].

{defi}

[Barthe et al. [2013]] Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$ be sub-distributions, $\varepsilon,\delta\in\mathbb{R}^{\geq 0}$ and $\mathrel{\mathcal{R}}$ be a binary relation over $A$ and $B$. An $(\varepsilon,\delta)$-approximate symmetric $1$-lifting of $\mu_{1}$ and $\mu_{2}$ for $\mathrel{\mathcal{R}}$ is a sub-distribution $\mu\in{\mathbb{D}}(A\times B)$ s.t.

1. (1)

$\pi_{1}^{\sharp}(\mu)\leq\mu_{1}$ and $\pi_{2}^{\sharp}(\mu)\leq\mu_{2}$; 2. (2)

$\Delta_{\varepsilon}(\mu_{1},\pi_{1}^{\sharp}(\mu))\leq\delta$ and $\Delta_{\varepsilon}(\mu_{2},\pi_{2}^{\sharp}(\mu))\leq\delta$; and 3. (3)

$\operatorname{supp}(\mu)\subseteq\mathrel{\mathcal{R}}$.

We will write $\mu_{1}\mathrel{\overline{\mathrel{\mathcal{R}}}^{(1)}_{\varepsilon,\delta}}\mu_{2}$ if there exists an $(\varepsilon,\delta)$-approximate symmetric $1$-lifting of $\mu_{1}$ and $\mu_{2}$ for $\mathrel{\mathcal{R}}$; the superscript $\cdot^{(1)}$ indicates that there is one witness for this lifting.

Theorem 22** (cf. the asymmetric result Theorem 21).**

For all binary relations $\mathrel{\mathcal{R}}$ over $A$ and $B$ and parameters $\varepsilon,\delta\geq 0$, we have $\mathrel{\overline{\mathrel{\mathcal{R}}}^{(1)}_{\varepsilon,\delta}}=\mathrel{\overline{\mathrel{\mathcal{R}}}^{(\star)}_{\varepsilon,\delta}}.$

Proof 6.1**.**

See Appendix, p. 7.5

+=

Proof 6.2** (Proof of ).**

Suppose that $(\mu_{L},\mu_{R})$ are witnesses to $\mu_{1}\mathrel{\overline{\mathrel{\mathcal{R}}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$. Define the witness $\eta\in{\mathbb{D}}(A\times B)$ as the point-wise minimum: $\eta(a,b)=\min(\mu_{L}(a,b),\mu_{R}(a,b))$. We will show that $\eta$ is a witness to $\mu_{1}\mathrel{\overline{\mathrel{\mathcal{R}}}^{(1)}_{\varepsilon,\delta}}\mu_{2}$.

The support condition follows from the support condition for $(\mu_{L},\mu_{R})$. The marginal conditions $\pi_{1}^{\sharp}(\eta)\leq\mu_{1}$ and $\pi_{2}^{\sharp}(\eta)\leq\mu_{2}$ also follow by the marginal conditions for $(\mu_{L},\mu_{R})$. The only thing to check is the distance condition. By the distance condition on $(\mu_{L},\mu_{R})$, there exist non-negative values $\delta(a,b)$ such that

[TABLE]

and $\sum_{a,b}\delta(a,b)\leq\delta$. So, $\mu_{R}(a,b)\geq\exp(-\varepsilon)(\mu_{L}(a,b)-\delta(a,b))$. Similarly, there are non-negative values $\delta^{\prime}(a,b)$ such that

[TABLE]

and $\sum_{a,b}\delta^{\prime}(a,b)\leq\delta$. So, $\mu_{L}(a,b)\geq\exp(-\varepsilon)(\mu_{R}(a,b)-\delta^{\prime}(a,b))$.

Now let $S\subseteq A$ be any subset. Then:

[TABLE]

The other marginal is similar. For any subset $T\subseteq B$ we have

[TABLE]

Thus, $\eta$ witnesses $\mu_{1}\mathrel{\overline{\mathrel{\mathcal{R}}}^{(1)}_{\varepsilon,\delta}}\mu_{2}$.

The other direction is more interesting. Let $\eta\in{\mathbb{D}}(A\times B)$ be the single witness to $\mathrel{\overline{\mathrel{\mathcal{R}}}^{(1)}_{\varepsilon,\delta}}$. By the distance conditions $\Delta_{\varepsilon}(\mu_{1},\pi_{1}^{\sharp}\eta)\leq\delta$ and $\Delta_{\varepsilon}(\mu_{2},\pi_{2}^{\sharp}\eta)\leq\delta$, there exist non-negative values $\delta(a)$ and $\delta^{\prime}(b)$ such that

[TABLE]

there is equality when $\delta(a)$ or $\delta^{\prime}(b)$ are strictly positive, and both $\sum_{a\in A}\delta(a)$ and $\sum_{b\in B}\delta^{\prime}(b)$ are at most $\delta$. Define two witnesses $\mu_{L}\in{\mathbb{D}}(A\times B^{\star}),\mu_{R}\in{\mathbb{D}}(A^{\star}\times B)$ as follows:

[TABLE]

(As usual, if any denominator is zero, we take the probability to be zero as well.)

The support condition follows from the support condition of $\eta$. The marginal conditions hold by definition. Note that all probabilities are non-negative. For instance in $\mu_{L}$, note that if $\delta(a)>0$ then $\mu_{1}(a)-\delta(a)=\exp(\varepsilon)\pi_{1}^{\sharp}\eta(a)\geq 0$ and hence

[TABLE]

assuming $\pi_{1}^{\sharp}\eta(a)>0$; if $\pi_{1}^{\sharp}\eta(a)=0$ then $\mu_{L}(a,\star)=0$. A similar argument shows that $\mu_{R}$ is non-negative.

So, it remains to check the distance bounds. Note that when $a,b\neq\star$, by definition $\mu_{L}(a,b)$ and $\mu_{R}(a,b)$ are both strictly positive or both equal to zero, and $\eta(a,b)$ is strictly positive or equal to zero accordingly. If $\mu_{L}(a,b),\mu_{R}(a,b),\eta(a,b)$ are all strictly positive, then we know

[TABLE]

We can also lower bound the ratios:

[TABLE]

for instance when $\delta(a)>0$ then the ratio is exactly equal to $\exp(\varepsilon)\geq 1$, and when $\delta(a)=0$ then the ratio is at least $1$ by the marginal property $\pi_{1}^{\sharp}\eta\leq\mu_{1}$. So we have $\mu_{L}(a,b)/\eta(a,b)$ and $\mu_{R}(a,b)/\eta(a,b)$ in $[1,\exp(\varepsilon)]$ when all distributions are strictly positive. Thus we always have

[TABLE]

We can also bound the mass on points $(a,\star)$. Let $S\subseteq A$ be any subset. $\overline{\mu_{R}}(S\times\{\star\})\leq\exp(\varepsilon)\overline{\mu_{L}}(S\times\{\star\})+\delta$ is clear. For the other direction:

[TABLE]

The mass at points $(\star,b)$ can be bounded in a similar way. Let $T\subseteq B$ be any subset. Then, $\overline{\mu_{L}}(\{\star\}\times T)\leq\exp(\varepsilon)\overline{\mu_{R}}(\{\star\}\times T)+\delta$ is clear. For the other direction:

[TABLE]

So $\Delta_{\varepsilon}(\overline{\mu_{L}},\overline{\mu_{R}})\leq\delta$ and $\Delta_{\varepsilon}(\overline{\mu_{R}},\overline{\mu_{L}})\leq\delta$ so we have witnesses to $\mu_{1}\mathrel{\overline{\mathrel{\mathcal{R}}}^{(\star)}_{\varepsilon,\delta}}\mu_{2}$. Hence, $\mathrel{\overline{\mathrel{\mathcal{R}}}^{(1)}_{\varepsilon,\delta}}=\mathrel{\overline{\mathrel{\mathcal{R}}}^{(\star)}_{\varepsilon,\delta}}$.

The main use of symmetric approximate liftings is to support richer composition results that only apply to symmetric adjacency relations.

{defi}

Let $\mathbb{R}^{\geq 0}_{2}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\mathbb{R}^{\geq 0}\times\mathbb{R}^{\geq 0}$ and let $(\mathbb{R}^{\geq 0}_{2})^{*}$ be the set of finite sequences over pairs of non-negative reals. A map $r:(\mathbb{R}^{\geq 0}_{2})^{*}\to\mathbb{R}^{\geq 0}_{2}$ is a symmetric DP-composition rule if for all sets $A,D$, symmetric adjacency relations $\phi\subseteq D\times D$, and families of functions $\{f_{i}:D\times A\to{\mathbb{D}}(A)\}_{i<n}$, the following implication holds: if for every initial value $a\in A$ and $i<n$, $f_{i}(-,a):D\to{\mathbb{D}}(A)$ is $(\varepsilon_{i},\delta_{i})$-differentially private w.r.t. $\phi$, then $F(-,a)$ is $(\varepsilon^{*},\delta^{*})$-differentially private w.r.t. $\phi$ and any initial value $a\in A$ where $F:(d,a)\mapsto(\bigcirc_{i<n}\mathrel{(f_{i}(d,-))^{\sharp}})({{\mathbb{1}}_{a}})$ is the $n$-fold composition of the functions $[f_{i}]_{i<n}$ and $(\varepsilon^{*},\delta^{*})\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}r([(\varepsilon_{i},\delta_{i})]_{i<n})$.

We have the following reduction, a symmetric version of Lemma 19.

Lemma 23**.**

Let $r:(\mathbb{R}^{\geq 0}_{2})^{*}\to\mathbb{R}^{\geq 0}_{2}$ be a symmetric DP-composition rule. Let $n\in\mathbb{N}$ and assume given two families of sets $\{A_{i}\}_{i\leq n}$ and $\{B_{i}\}_{i\leq n}$, together with a family of binary relations $\{\mathcal{R}(i)\subseteq A_{i}\times B_{i}\}_{i\leq n}$. Fix two families of functions $\{g_{i}:A_{i}\to{\mathbb{D}}(A_{i+1})\}_{i<n}$ and $\{h_{i}:B_{i}\to{\mathbb{D}}(B_{i+1})\}_{i<n}$ s.t. for any $i<n$ and $(a,b)\in\mathcal{R}(i)$ we have:

1. (1)

$g_{i}(a)\mathrel{\mathrel{\overline{\mathcal{R}(i+1)}^{(\star)}_{\varepsilon_{i},\delta_{i}}}}h_{i}(b)$* for some parameters $\varepsilon_{i},\delta_{i}\geq 0$, and* 2. (2)

$g_{i}(a)$* and $h_{i}(b)$ are proper distributions.*

Then for $(a_{0},b_{0})\in\mathcal{R}(0)$, there exists a symmetric $\star$-lifting

[TABLE]

where $G:A_{0}\to{\mathbb{D}}(A_{n})$ and $H:B_{0}\to{\mathbb{D}}(B_{n})$ are the $n$-fold compositions of $[g_{i}]_{i\leq n}$ and $[h_{i}]_{i\leq n}$ respectively—i.e. $G(a)\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}(\bigcirc_{i<n}\mathrel{g_{i}^{\sharp}})({{\mathbb{1}}_{a}})$ and $H(b)\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}(\bigcirc_{i<n}\mathrel{h_{i}^{\sharp}})({{\mathbb{1}}_{b}})$ and $(\varepsilon^{*},\delta^{*})\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}r([(\varepsilon_{i},\delta_{i})]_{i<n})$.

Proof 6.3**.**

Essentially the same as the proof of Lemma 19. Let $D=\{\mathit{tt},\mathit{ff}\}$ as before, and take $\overline{\phi}=\{(\mathit{tt},\mathit{ff}),(\mathit{ff},\mathit{tt})\}$ be a binary relation on $D$.

For every $i<n$ and $(a_{i},b_{i})\in\mathcal{R}(i)$, the definition of symmetric $\star$-lifting gives two distributions $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}[a_{i},b_{i}],\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}[a_{i},b_{i}]$ witnessing $g_{i}(a)\mathrel{\mathrel{\overline{\mathcal{R}(i+1)}^{(\star)}_{\varepsilon_{i},\delta_{i}}}}$. We define the same maps $f_{i}:D\times(A^{\star}\times B^{\star})\to{\mathbb{D}}(A^{\star}\times B^{\star})$ as before:

[TABLE]

for $(a_{i},b_{i})\in\mathcal{R}(i)$; otherwise, $f_{i}(-,(a,b))=0$.

Compared to proof of Lemma 19, the crucial difference is that since we have witnesses to a symmetric approximate lifting, the resulting maps $f_{i}(-,(a,b)):D\to{\mathbb{D}}(A^{\star}\times B^{\star})$ are $(\varepsilon_{i},\delta_{i})$-differentially private with respect to the symmetric relation $\overline{\phi}$, not just the asymmetric relation $\phi$. Hence, we may apply the symmetric DP-composition rule $r$ and conclude as before.

With this reduction we hand, we can generalize the advanced composition theorem from differential privacy to $\star$-liftings.

Theorem 24** (Advanced composition [Dwork et al., 2010]).**

Consider a symmetric adjacency relation $\phi$ on databases $D$. Let $f_{i}:D\times A\to{\mathbb{D}}(A)$ be a sequence of $n$ functions, such that for every $a\in A$ the functions $f_{i}(-,a):D\to{\mathbb{D}}(A)$ are $(\varepsilon,\delta)$-differentially private with respect to $\phi$. Then, for every $a\in A$ and $\omega\in(0,1)$, running $f_{1},\dots,f_{n}$ in sequence is $(\varepsilon^{*},\delta^{*})$-differentially private for

[TABLE]

Corollary 25**.**

Let $n$ be a natural number, $\varepsilon,\delta\geq 0$, and $\omega\in(0,1)$ be real parameters. Suppose we have:

1. (1)

sets $\{A_{i}\}_{i},\{B_{i}\}_{i}$ with $i$ ranging from $0,\dots,n$; 2. (2)

relations $\{\mathcal{R}(i)\}_{i}$ on $A_{i}$ and $B_{i}$ with $i$ ranging from $0,\dots,n$; and 3. (3)

functions $\{f_{i}:A_{i}\to{\mathbb{D}}(A_{i+1})\}_{i},\{g_{i}:B_{i}\to{\mathbb{D}}(B_{i+1})\}_{i}$ with $i$ ranging from $0,\dots,n-1$

such that for all $(a,b)\in\mathcal{R}(i)$, we have

[TABLE]

and $f_{i}(a),g_{i}(b)$ proper distributions. Then, there is an approximate lifting of the compositions:

[TABLE]

for every $(a_{0},b_{0})\in\mathcal{R}(0)$, where $F:A_{0}\to{\mathbb{D}}(A_{n})$ and $G:B_{0}\to{\mathbb{D}}(B_{n})$ are the $n$-fold (Kleisli) compositions of $\{f_{i}\}$ and $\{g_{i}\}$ respectively, and the lifting parameters are:

[TABLE]

Proof 6.4**.**

By the advanced composition theorem for differential privacy (Theorem 24), the map $r([(\varepsilon,\delta)]_{i<n})\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}(\varepsilon^{\prime},\delta^{\prime})$ is a symmetric DP composition rule. So, we can conclude by Lemma 23.

7. $\star$-Lifting for $f$-Divergences

The definition of $\star$-lifting can be extended to lifting constructions based on general $f$-divergences, as previously proposed by Barthe and Olmedo [2013], Olmedo [2014]. Roughly, a $f$-divergence is a function $\Delta_{f}(\mu_{1},\mu_{2})$ that measures the difference between two probability distributions $\mu_{1}$ and $\mu_{2}$. Much like we generalized the definition for $(\varepsilon,\delta)$-liftings, we can define $\star$-lifting with $f$-divergences. Let us first formally define $f$-divergences. We denote by $\mathcal{F}$ the set of non-negative convex functions vanishing at $1$: $\mathcal{F}=\{f:\mathbb{R}^{\geq 0}\rightarrow\mathbb{R}^{\geq 0}\mid f(1)=0\}$. We also adopt the following notational conventions: $0\cdot f(0/0)\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}0$, and $0\cdot f(x/0)\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}x\cdot\lim_{t\to 0^{+}}t\cdot f(1/t)$; we write $L_{f}$ for the limit.

{defi}

Given $f\in\mathcal{F}$ , the $f$-divergence $\Delta_{f}(\mu_{1},\mu_{2})$ between two distributions $\mu_{1}$ and $\mu_{2}$ in ${\mathbb{D}}(A)$ is defined as

[TABLE]

Examples of $f$-divergences include statistical distance ($f(t)=\frac{1}{2}\left|t-1\right|$), Kullback-Leibler divergence ($f(t)=t\ln(t)-t+1$),333The additional term $-t+1$ extends the classical definition of KL-divergence to sub-distributions [Barthe and Olmedo, 2013]. and Hellinger distance $(f(t)=\frac{1}{2}(\sqrt{t}-1)^{2}$).

{defi}

[$\star$-lifting for $f$-divergences] Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$ be distributions, $\mathrel{\mathcal{R}}$ be a binary relation over $A$ and $B$, and $f\in\mathcal{F}$. An $(f;\delta)$-approximate lifting of $\mu_{1}$ and $\mu_{2}$ for $\mathrel{\mathcal{R}}$ is a pair of distributions $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}\in{\mathbb{D}}(A\times B^{\star})$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}\in{\mathbb{D}}(A^{\star}\times B)$ s.t.

• •

$\pi_{1}^{\sharp}(\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})=\mu_{1}$ and $\pi_{2}^{\sharp}(\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})=\mu_{2}$;

• •

$\operatorname{supp}({\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}_{|A\times B}),\operatorname{supp}({\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}_{|A\times B})\subseteq\mathrel{\mathcal{R}}$; and

• •

$\Delta_{f}(\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}},\overline{\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}})\leq\delta$,

where $\overline{\eta_{\bullet}}$ is the canonical lifting of $\eta_{\bullet}$ to $A^{\star}\times B^{\star}$. We will write: $\mu_{1}\mathrel{R^{(\star)}_{f;\delta}}\mu_{2}$ if there exists an $(f;\delta)$-approximate lifting of $\mu_{1}$ and $\mu_{2}$ for $\mathrel{\mathcal{R}}$.

$\star$-liftings for certain $f$-divergences compose sequentially.

Lemma 26**.**

Suppose $f$ corresponds to statistical distance, Kullback-Leibler, or Hellinger distance. For $i\in\{1,2\}$, let $\mu_{i}\in{\mathbb{D}}(A_{i})$ and $\eta_{i}:A_{i}\to{\mathbb{D}}(B_{i})$. Let $\mathrel{\mathcal{R}}$ (resp. $\mathrel{\mathcal{S}}$) be a binary relation over $A_{1}$ and $A_{2}$ (resp. over $B_{1}$ and $B_{2}$). If $\mu_{1}\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{f;\delta}}\mu_{2}$ for some $\delta\geq 0$ and for any $(a_{1},a_{2})\in\mathrel{\mathcal{R}}$ we have $\eta_{1}(a_{1})\mathrel{\mathrel{\mathcal{S}}^{(\star)}_{f;\delta^{\prime}}}\eta_{2}(a_{2})$ for some $\delta^{\prime}\geq 0$, then

[TABLE]

Proof 7.1**.**

Essentially the same as the proof of Lemma 19, lifting known composition results for these $f$-divergences (namely, Barthe and Olmedo [2013, Proposition 5]).

Much like the $\star$-liftings we saw before, $\star$-liftings for $f$-divergences have witness distributions with support determined by the support of $\mu_{1}$ and $\mu_{2}$ (cf. Lemma 4).

Lemma 27**.**

Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$ be distributions such that $\mu_{1}\mathrel{R^{(\star)}_{f;\delta}}\mu_{2}$ . Then, there are witnesses with support contained in $\operatorname{supp}(\mu_{1})^{\star}\times\operatorname{supp}(\mu_{2})^{\star}$.

Proof 7.2**.**

See Appendix, p. 7.5

+=

Proof 7.3** (Proof of ).**

Let $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ be any pair of witnesses to the approximate lifting. We will construct witnesses $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}},\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ with the desired support. For ease of notation, let $S_{i}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\operatorname{supp}(\mu_{i})$ for $i\in\{1,2\}$. Define:

[TABLE]

Evidently, $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ have support in $S_{1}^{\star}\times S_{2}^{\star}$. Additionally, it is straightforward to check that $\pi_{1}^{\sharp}(\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})=\pi_{1}^{\sharp}(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})=\mu_{1}$ and $\pi_{2}^{\sharp}(\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})=\pi_{2}^{\sharp}(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})=\mu_{2}$ so $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ have the desired marginals.

It only remains to check the distance condition. We can compute:

[TABLE]

Now, note that for all $b^{\prime}\in B^{\star}\mathrel{-}S_{2}$, we know $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}(a,b^{\prime})=0$. Similarly, for all $a^{\prime}\in A^{\star}\mathrel{-}S_{1}$, we know $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}(a^{\prime},b)=0$. Hence, the last line is equal to

[TABLE]

Thus, $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ witness the desired $\star$-lifting.

Finally, the mapping property from Lemma 13 holds also for these $\star$-liftings. While the proof of Lemma 13 relies on the equivalence for Sato’s definition, there is no such equivalence (or definition) for general $f$-divergences. Therefore, we must work directly with the witnesses of the approximate lifting.

Lemma 28**.**

Let $\mu_{1}\in{\mathbb{D}}(A_{1})$, $\mu_{2}\in{\mathbb{D}}(A_{2})$, $g_{1}:A_{1}\to B_{1}$, $g_{2}:A_{2}\to B_{2}$ and $\mathrel{\mathcal{R}}$ a binary relation on $B_{1}$ and $B_{2}$. Let $\mathrel{\mathcal{S}}$ such that $a_{1}\mathrel{\mathrel{\mathcal{S}}}a_{2}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{\iff}}}g_{1}(a_{1})\mathrel{\mathcal{R}}g_{2}(a_{2})$. Then

[TABLE]

Proof 7.4**.**

See Appendix, p. 7.5

+=

Proof 7.5** (Proof of ).**

For the reverse direction, take the witnesses $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}},\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}\in{\mathbb{D}}(A^{\star}\times A^{\star})$ and define witnesses $\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\mathrel{(g_{1}^{\star}\times g_{2}^{\star})^{\sharp}}(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})$ and $\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\mathrel{(g_{1}^{\star}\times g_{2}^{\star})^{\sharp}}(\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})$, where $g_{1}^{\star}\times g_{2}^{\star}$ takes a pair $(a_{1},a_{2})$ to the pair $(g_{1}(a_{1}),g_{2}(a_{2}))$ and maps $\star$ to $\star$. The support and marginal requirements are clear. The only thing to check is the distance condition, but this follows from monotonicity of $f$-divergences—under the mapping $g_{1}^{\star}\times g_{2}^{\star}:A^{\star}\times A^{\star}\to B^{\star}\times B^{\star}$, the $f$-divergence can only decrease (see, e.g., Csiszár and Shields [2004]).

For the forward direction, let $\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}},\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}\in{\mathbb{D}}(B^{\star}\times B^{\star})$ be the witnesses to the second lifting. By Lemma 27, we may assume without loss of generality that $\operatorname{supp}(\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})$ and $\operatorname{supp}(\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}})$ are contained in

[TABLE]

We aim to construct a pair of witnesses $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}},\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}\in{\mathbb{D}}(A^{\star}\times A^{\star})$ to the first lifting. The basic idea is to define $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ based on equivalence classes of elements in $A$ mapping to a particular $b\in B$, and then smooth out the probabilities within each equivalence class. To begin, for $a\in A$, define $[a]_{g}\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}g^{-1}(g(a))$ and $\alpha_{i}(a)\mathrel{\stackrel{{\scriptstyle\scriptscriptstyle\triangle}}{{=}}}\textstyle\Pr_{\mu_{i}}[\{a\}\mid[a]_{g_{i}}]$. We take $\alpha_{i}(a)=0$ when $\mu_{i}([a]_{f_{i}})=0$, and we let $\alpha_{i}(\star)=0$. We define $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$ and $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ as

[TABLE]

where

[TABLE]

The support and marginal conditions follow from the support and marginal conditions of $\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}$, $\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$. For instance:

[TABLE]

In the last line, we replace the sum over $b_{2}\in g_{2}(A^{\star})$ with a sum over $b_{2}\in B^{\star}$; this holds since the support of $\mathrel{g_{2}^{\sharp}}(\mu_{2})$ is contained in $g_{2}(A)$, so we can assume that $\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}(a,b_{2})=0$ for all $b_{2}$ outside of $g_{2}(A^{\star})$. Then, we can conclude by the marginal condition $\pi_{1}^{\sharp}(\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}})=\mathrel{g_{1}^{\sharp}}(\mu_{1})$. The second marginal is similar.

We now check the distance condition $\Delta_{f}(\overline{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}},\overline{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}})\leq\delta$. We can split the $f$-divergence into $\Delta_{f}(\overline{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}},\overline{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}})=P_{0}+P_{1}+P_{2}+P_{3}$, where

[TABLE]

We will handle each term separately. Evidently $P_{0}=0$. For $P_{1}$, we have

[TABLE]

where the sets $S^{=0}$ and $S^{\neq 0}$ are

[TABLE]

By further rearranging,

[TABLE]

The final equality is because without loss of generality, we can assume (by Lemma 27) that $\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}},\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ are zero outside of the support of $\mathrel{g_{1}^{\sharp}}(\mu_{1})$ and $\mathrel{g_{2}^{\sharp}}(\mu_{2})$, which have support contained in $(g_{1}\times g_{2})(A\times A)$.

The remaining two terms $P_{2}$ and $P_{3}$ are simpler to bound. For $P_{2}$, note that $\overline{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}(a,\star)=0$ for all $a\in A$. Thus:

[TABLE]

where the last equality is because $\overline{\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}}(b,\star)=0$ for all $b\in B$.

Similarly for $P_{3}$, using $\overline{\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}(\star,a)=\overline{\nu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}}(\star,b)=0$ for all $a\in A$ and $b\in B$, we have:

[TABLE]

Putting everything together, we conclude

[TABLE]

by assumption, so $\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}},\mu_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}$ witness the desired approximate lifting.

8. Conclusion

We have proposed a new definition of approximate lifting that unifies all known existing constructions and satisfies an approximate variant of Strassen’s theorem. Our notion is useful both to simplify the soundness proof of existing program logics and to strengthen some of their proof rules.

Subsequent to the original publication of this work, researchers have explored two extensions of $\star$-liftings. First, Albarghouthi and Hsu [2018] develop variable approximate liftings, a refinement of the $(\varepsilon,0)$-approximate liftings where the $\varepsilon$ parameter is real-valued function $A\times B\to\mathbb{R}^{\geq 0}$ and the distance condition on witnesses is generalized to

[TABLE]

In effect, the approximation parameters may vary over pairs of samples instead of being a uniform upper bound. This refinement allows capturing more precise approximation bounds, in some cases simplifying proofs of differential privacy. An $(\varepsilon,\delta)$ version of variable approximate lifting is currently not known.

Second, Sato et al. [2018] explore 2-liftings in the continuous case modeling differential privacy and $f$-divergences, but also relaxations of differential privacy based on Rényi divergences [Bun and Steinke, 2016, Mironov, 2017]. These 2-liftings subsume $\star$-liftings; a continuous analogue of the approximate Strassen theorem is strongly believed to hold but remains to be shown.

We see at least two important directions for future work. First, adapting existing program logics (in particular, apRHL [Barthe et al., 2013]) to use $\star$-liftings, and formalizing examples that were out of reach of previous systems. Second, symmetric $\star$-liftings seem to be an important notion—for instance, the advanced composition theorem of differential privacy [Dwork et al., 2010] applies to these liftings—but only existential versions of the definition are currently known. A universal definition, similar to Sato’s definition for asymmetric liftings, would give more evidence that symmetric liftings are indeed a mathematically interesting abstraction, and also give a more convenient route to constructing such liftings.

Acknowledgments

We thank the anonymous reviewers for their helpful suggestions. This work was partially supported by a grant from the NSF (TWC-1513694), a grant from the Simons Foundation (#360368 to Justin Hsu), and the ERC Starting Grant ProFoundNet (#679127).

Appendix A Detailed Proofs

In the proofs, we will sometimes refer to the witnesses of a $\star$-lifting.

{nota}

Let $\mu_{1}\in{\mathbb{D}}(A)$ and $\mu_{2}\in{\mathbb{D}}(B)$ be sub-distributions, $\varepsilon,\delta\in\mathbb{R}^{+}$ and $\mathrel{\mathcal{R}}$ be a binary relation over $A$ and $B$. If two distributions $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleleft}}\in{\mathbb{D}}(A\times B^{\star})$ and $\eta_{\mspace{-1.5mu}{\scriptscriptstyle\vartriangleright}}\in{\mathbb{D}}(A^{\star}\times B)$ are witnesses to the $\star$-lifting $\mu_{1}\mathrel{\mathrel{\mathrel{\mathcal{R}}^{(\star)}_{\varepsilon,\delta}}}\mu_{2}$, then we write:

[TABLE]

\prooftoks\prooftoks