Collaborative Mean Estimation over Intermittently Connected Networks with Peer-To-Peer Privacy

TL;DR

This paper introduces a differentially private collaborative algorithm for distributed mean estimation over intermittently connected networks, balancing privacy and accuracy through local consensus among nodes.

Contribution

It proposes a novel differentially private algorithm for distributed mean estimation that optimally balances collaboration benefits and privacy risks in intermittent networks.

Findings

The algorithm achieves the optimal privacy-accuracy tradeoff.

Numerical simulations confirm theoretical results.

Collaboration improves estimation accuracy under privacy constraints.

Abstract

This work considers the problem of Distributed Mean Estimation (DME) over networks with intermittent connectivity, where the goal is to learn a global statistic over the data samples localized across distributed nodes with the help of a central server. To mitigate the impact of intermittent links, nodes can collaborate with their neighbors to compute local consensus which they forward to the central server. In such a setup, the communications between any pair of nodes must satisfy local differential privacy constraints. We study the tradeoff between collaborative relaying and privacy leakage due to the additional data sharing among nodes and, subsequently, propose a novel differentially private collaborative algorithm for DME to achieve the optimal tradeoff. Finally, we present numerical simulations to substantiate our theoretical findings.

Figures

Equations

$${\tilde{\mathbf{x}}}_{ji}={\tau }_{ji}({\alpha }_{ji}{\mathbf{x}}_j+{\mathbf{n}}_{ji})$$

$${\tilde{\mathbf{x}}}_i=\sum _{j\in [n]}{\tau }_{ji}({\alpha }_{ji}{\mathbf{x}}_j+{\mathbf{n}}_{ji}).$$

$$\sum _{j\in [n]}{p}_j{p}_{ij}{\alpha }_{ij}=1,$$

$$\mathbb{E}\Vert \widehat{\overline{\mathbf{x}}}-\mathbf{x}{\Vert }_2^2\le {\mathrm{R}}^2{\sigma }_{\mathrm{tv}}^2(\mathbf{p},\mathbf{P},\mathbf{A})+{\sigma }_{\mathrm{pr}}^2(\mathbf{p},\mathbf{P},\sigma ),$$

$${\sigma }_{\mathrm{tv}}^2(\mathbf{p},\mathbf{P},\mathbf{A})\triangleq \frac{1}{n^2}[\sum _{i,j,l\in [n]}{p}_j(1-p_j)p_{ij}{p}_{lj}{\alpha }_{ij}{\alpha }_{lj}$$

$$+\sum _{i,j\in [n]}{p}_{ij}{p}_j(1-p_{ij}){\alpha }_{ij}^2+\sum _{i,l\in [n]}{p}_i{p}_l({\mathrm{E}}_{\{i,l\}}-p_{il}{p}_{li}){\alpha }_{li}{\alpha }_{il}],$$

$${\sigma }_{\mathrm{pr}}^2(\mathbf{p},\mathbf{P},\sigma )\triangleq \frac{1}{n^2}\sum _{i,j\in [n]}{p}_j{p}_{ij}{\sigma }^{2}d.$$

$${ϵ}_{ij}=\{\begin{array}{l}{\left[2\ln \left(\frac{1.25}{{\delta }_{ij}}\right)\right]}^{\frac{1}{2}}\frac{2{\alpha }_{ij}\mathrm{R}}{\sigma }\text{ if }{p}_{ij}>0,\text{ and, }\\ 0\text{ if }{p}_{ij}=0,\end{array}$$

$$\mathrm{Pr}\left({\tilde{\mathbf{x}}}_{ij}\in \mathcal{S}\mid {\mathbf{x}}_i\in {\mathcal{D}}_i\right)\le e^{{ϵ}_{ij}}\mathrm{Pr}\left({\tilde{\mathbf{x}}}_{ij}\in \mathcal{S}\mid {\mathbf{x}}_i^{\prime }\in {\mathcal{D}}_i\right)+p_{ij}{\delta }_{ij},$$

$$\underset{\mathbf{A},\sigma }{\min }{\mathrm{R}}^2{\sigma }_{\mathrm{tv}}^2(\mathbf{p},\mathbf{P},\mathbf{A})+{\sigma }_{\mathrm{pr}}^2(\mathbf{p},\mathbf{P},\sigma )$$

$$\text{s.t.: }{\alpha }_{ij}\ge 0,\forall i,j\in [n],\text{(non-negative weights)}$$

$$\sum _{j\in [n]}{p}_j{p}_{ij}{\alpha }_{ij}=1,\forall i\in [n],\text{(unbiasedness)}$$

$${\left[2\ln \left(\frac{1.25}{{\underline{\delta }}_{ij}}\right)\right]}^{\frac{1}{2}}\frac{2{\alpha }_{ij}\mathrm{R}}{\sigma }\le {\underline{ϵ}}_{ij}\forall i,j\in [n],\text{(privacy)}$$

$$\sigma \ge 0.\text{(privacy)}.$$

$${\sigma }_{\mathrm{thr}}\triangleq 2\mathrm{R}{\max }_{i\in [n]}{\left(\sum _{j\in [n]}{p}_j{p}_{ij}{\underline{ϵ}}_{ij}[2\ln (\frac{1.25}{{\underline{\delta }}_{ij}}){]}^{-\frac{1}{2}}\right)}^{-1}\ge 0.$$

$$\underset{\sigma }{\min }\frac{1}{n^2}\sum _{i,j\in [n]}{p}_j{p}_{ij}{\sigma }^{2}d$$

$$\text{s.t.: }{\left[2\ln \left(\frac{1.25}{{\underline{\delta }}_{ij}}\right)\right]}^{\frac{1}{2}}\frac{2{\alpha }_{ij}\mathrm{R}}{\sigma }\le {\underline{ϵ}}_{ij}\forall i,j\in [n],$$

$$\sigma \ge {\sigma }_{\mathrm{thr}}.$$

$$\sigma$$

$$\underset{\mathbf{A}}{\min }{\mathrm{R}}^2{\sigma }_{\mathrm{tv}}^2(\mathbf{p},\mathbf{P},\mathbf{A})$$

$$\text{s.t.: }{\alpha }_{ij}\ge 0,\forall i,j\in [n],\sum _{j\in [n]}{p}_j{p}_{ij}{\alpha }_{ij}=1,\forall i\in [n],$$

$${\left[2\ln \left(\frac{1.25}{{\underline{\delta }}_{ij}}\right)\right]}^{\frac{1}{2}}\frac{2{\alpha }_{ij}\mathrm{R}}{\sigma }\le {\underline{ϵ}}_{ij}\forall i,j\in [n].$$

$$\underset{\mathbf{A}}{\min }{\mathrm{R}}^2{\overline{\sigma }}_{\mathrm{tv}}^2(\mathbf{p},\mathbf{P},\mathbf{A})\text{ s.t. the same constraints as }\text{eqref}eq:weigh{t}_{g}ive{n}_{n}ois{e}_{o}p{t}_{p}ro{b}_{u}ncorrelated,$$

$${\overline{\sigma }}_{\mathrm{tv}}^2(\mathbf{p},\mathbf{P},\mathbf{A})\triangleq \frac{1}{n^2}[\sum _{i,j,l\in [n]}{p}_j(1-p_j)p_{ij}{p}_{lj}{\alpha }_{ij}{\alpha }_{lj}$$

$$+\sum _{i,j\in [n]}{p}_{ij}{p}_j(1-p_{ij}){\alpha }_{ij}^2+\sum _{i,l\in [n]}{p}_i{p}_l({\mathrm{E}}_{\{i,l\}}-p_{il}{p}_{li}){\alpha }_{il}^2],$$

$${\mathbf{A}}_i^{(\ell )}=\{\begin{array}{l}{\hat{\mathbf{A}}}_i^{(\ell )}\text{ if }i=\ell \mathrm{mod}n+n\text{mathds}{1}_{\{\ell \mathrm{mod}n=0\}},\\ {\mathbf{A}}_i^{(\ell -1)}\text{ otherwise},\end{array}$$

$${\overline{\alpha }}_{ij}({\lambda }_i)\triangleq {\left(\frac{-2(1-p_j){\sum }_{l\in [n]:l\ne i}{p}_{lj}{\alpha }_{lj}^{(\ell -1)}+{\lambda }_i}{2[\left(1-p_j{p}_{ij}\right)+p_i(E_{\{i,j\}}/p_{ij}-p_{ji})]}\right)}^{+}.$$

$${\hat{\alpha }}_{ij}=\min \left\{{\tilde{\alpha }}_{ij}({\lambda }_i),{\tilde{w}}_{ij}\right\},$$

$${\tilde{\alpha }}_{ij}({\lambda }_i)=\{\begin{array}{ll}{\overline{\alpha }}_{ij}({\lambda }_i)& \text{ if }{p}_j{p}_{ij}>0,\\ 0& \text{ if }{p}_j{p}_{ij}=0.\end{array}$$

$${\tilde{\alpha }}_{ij}({\lambda }_i)=\{\begin{array}{ll}{\overline{\alpha }}_{ij}({\lambda }_i)& \text{ if }{p}_j{p}_{ij}\in (0,1),\\ 0& \text{ if }{p}_j{p}_{ij}=0.\end{array}$$

$${\hat{\alpha }}_{ij}^{(\ell )}=\{\begin{array}{ll}1& \text{ if }j=i,\\ 0& \text{ otherwise}.\end{array}$$

$${\tilde{\alpha }}_{ij}({\lambda }_i)$$

$${-2p_i(E_{\{i,j\}}/p_{ij}-p_{ji}){\alpha }_{ji}^{(\ell -1)}+{\lambda }_i))}^{+}.$$

$$\mathrm{Pr}({\mathcal{M}}_i(x)\in \mathcal{S})\le e^{{ϵ}_i}\mathrm{Pr}({\mathcal{M}}_i(x^{\prime })\in \mathcal{S})+{\delta }_i.$$

$$\mathcal{M}(X)\triangleq f(X)+\mathcal{N}(0,{\sigma }^2\mathbf{I}).$$

$$ϵ=\frac{{\Delta }_f}{\sigma }\sqrt{2\log \frac{1.25}{\delta }}.$$

Full text

I Introduction

Distributed Mean Estimation (DME) is a fundamental statistical problem that arises in several applications, such as model aggregation in federated learning [1], distributed K-means clustering [2], distributed power iteration [3], etc. DME presents several practical challenges, which prior research [4, 5, 6, 7, 8] has considered, including the problem of straggler nodes, where nodes cannot send their data to the parameter server (PS). Typically, there are two types of stragglers: $\rm(i)$ computation stragglers, in which nodes cannot finish their local computation within a deadline, and $\rm(ii)$ communication stragglers, in which nodes cannot transmit their updates due to communication blockage [9, 10, 11, 12, 13, 14]. The problem of communication stragglers can be solved by relaying the updates/data to the PS via neighboring nodes. This approach was proposed and analyzed in [15, 16, 17], where it was shown that the proposed collaborative relaying scheme can be optimized to reduce the expected distance to optimality, both for DME [15] and federated learning [16, 17].

While the works [16, 17, 15] show that collaborative relaying reduces the expected distance to optimality, exchanging the individual data across nodes incurs privacy leakage caused by the additional estimates that are shared among the nodes. Nonetheless, this potential breach of privacy has not been addressed in the aforementioned works. To mitigate the privacy leakage in DME, we require a rigorous privacy notion. Within the context of distributed learning, local differential privacy (LDP) [18] has been adopted as a gold standard notion of privacy, in which a user can perturb and disclose a sanitized version of its data to an untrusted server. LDP ensures that the statistics of the user’s output observed by adversaries are indistinguishable regardless of the realization of any input data. In this paper, we focus on the node-level LDP where the neighboring nodes, as well as any eavesdropper that can observe the local node-node transmissions during collaborations, cannot infer the realization of the user’s data.

There has been extensive research into the design of distributed learning algorithms that are both communication efficient and private (see [19] for a comprehensive survey and references therein). It is worth noting that LDP requires a significant amount of perturbation noise to ensure reasonable privacy guarantees. Nonetheless, the amount of perturbation noise can be significantly reduced by considering the intermittent connectivity of nodes in the learning process [20]. The intermittent connectivity in DME amplifies the privacy guarantees; it provides a boosted level of anonymity due to partial communication with the server. Various random node participation schemes have been proposed to further improve the utility-privacy tradeoff in distributed learning, such as Poisson sampling [21], importance sampling [22, 23], and sampling with/without replacement [20]. In addition, Balle et al. investigated in [24], the privacy amplification in federated learning via random check-ins and showed that the privacy leakage scales as $O(1/\sqrt{n})$, where $n$ is the number of nodes. In other words, random node participation reduces the amount of noise required to achieve the same levels of privacy that are achieved without sampling.

So far, works in the privacy literature, such as [18, 19, 20, 21, 22, 23, 24], have not considered intermittent connectivity along with collaborative relaying, where nodes share their local updates to mitigate the randomness in network connectivity [16, 17, 15]. Thus, this paper aims to close this theoretical gap. To this end, we first show that there exists a tradeoff between collaborative relaying and privacy leakage due to data sharing among nodes for DME under intermittent connectivity assumption. We introduce our system model and proposed algorithm in §II, followed by its utility (MSE) and privacy analyses in §III and §IV respectively. We quantify this tradeoff by formulating it as an optimization problem and solve it approximately due to its non-convexity. Finally, we demonstrate the efficacy of our private collaborative algorithm through numerical simulations.

II System Model for Private Collaboration

Consider a distributed system with $n$ nodes, each having a vector $\mathbf{x}_{i}\in\mathbb{R}^{d}$, $\lVert\mathbf{x}_{i}\rVert_{2}\leq{\rm R}$ for some known ${\rm R}>0$. The nodes communicate with a parameter server (PS), as well as with each other over intermittent links with the goal of estimating their mean, $\overline{\mathbf{x}}\triangleq\frac{1}{n}\sum_{i=1}^{n}\mathbf{x}_{i}$ at the PS (Fig. 1). For any estimate $\widehat{\overline{\mathbf{x}}}$ of the mean, the evaluation metric for any estimate is the mean-square error (MSE), given by ${\cal E}\triangleq\mathbb{E}\lVert\widehat{\overline{\mathbf{x}}}-\overline{\mathbf{x}}\rVert_{2}^{2}$.

II-A Communication Model

As shown in Fig. 1, node $i$ can communicate with the PS with a probability $p_{i}$, with the link modeled using a Bernoulli random variable $\tau_{i}\sim{\rm Ber}(p_{i})$. Similarly, node $i$ can communicate with another node $j$ with probability $p_{ij}$, i.e., $\tau_{ij}\sim{\rm Ber}(p_{ij})$. The links between different node pairs are assumed to be statistically independent, i.e., $\tau_{i}\perp\tau_{j}$ for $i\neq j$, $\tau_{ij}\perp\tau_{ml}$ for $(i,j)\neq(m,l)$, $(j,i)\neq(m,l)$, and $\tau_{ij}\perp\tau_{l}$ for $i,j,l\in[n]$. The correlation due to channel reciprocity between a pair of nodes $i,j\in[n]$ is denoted by ${\rm E}_{\{i,j\}}\equiv\mathbb{E}[\tau_{ij}\tau_{ji}]$. We assume that $\mathrm{E}_{\{i,j\}}\geq p_{ij}p_{ji}$, i.e., $\mathbb{P}(\tau_{ij}=1|\tau_{ji}=1)\geq\mathbb{P}(\tau_{ij}=1)$ Furthermore, $p_{ii}=1\;\forall\;i\in[n]$, and if node $i$ can never transmit to $j$, we set $p_{ij}=0$. We denote $\mathbf{p}\equiv(p_{1},\ldots,p_{n})$ and $\mathbf{P}\equiv(p_{ij})_{i,j\in[n]}\in[0,1]^{n\times n}$.

II-B Privacy Model

The nodes are assumed to be honest but curious. They are honest because they faithfully compute the aggregate of the signals received, however, they are curious as they might be interested in learning sensitive information about nodes. Each node uses a local additive noise mechanism to ensure the privacy of its transmissions to neighboring nodes. We consider local privacy constraints, wherein node $i$ trusts another node $j$ to a certain extent and hence, randomizes its own data accordingly when sharing with node $j$ using a synthetic Gaussian noise (see [18]) to respect the privacy constraint while maintaining utility. We present a refresher on differential privacy and Gaussian mechanism in App. A.

II-C Private Collaborative Relaying for Mean Estimation

We now introduce our algorithm, PriCER: Private Collaborative Estimation via Relaying. PriCER is a two-stage semi-decentralized algorithm for estimating the mean. In the first stage, each node $j\in[n]$ sends a scaled and noise-added version of its data to a neighboring node $i\in[n]$ over the intermittent link $\tau_{ji}$. The transmitted signal is given by,

[TABLE]

Here, $\alpha_{ji}\geq 0$ is the weight used by node $j$ while sending to node $i$, and $\mathbf{n}_{ji}\sim{\cal N}(\mathbf{0},\sigma^{2}\mathbf{I}_{d})$ is the multivariate Gaussian privacy noise added by node $j$. Here, $\sigma^{2}$ is the variance of each coordinate, and $\mathbf{I}_{d}\in\mathbb{R}^{d\times d}$ is the identity matrix. We denote the weight matrix by $\mathbf{A}\equiv(\alpha_{ij})_{i,j\in[n]}$. Consequently, node $i$ computes the local aggregate of all received signals as,

[TABLE]

We quantify our privacy guarantees using the well-established notion of differential privacy [18]. By observing $\widetilde{\mathbf{x}}_{ji}$, node $i$ should not be able to distinguish between the events when node $j$ contains the data $\mathbf{x}_{j}$ versus when it contains some other data $\mathbf{x}_{j}^{\prime}$. In other words, we are interested in protecting the local data of node $j$ from a (potentially untrustworthy) neighboring node $i$. We assume that the privacy noise added by different nodes are uncorrelated, i.e., $\mathbb{E}[\mathbf{n}_{il}^{\top}\mathbf{n}_{jm}]=0$ for all $i,j,l,m\in[n]$ as long as $i,j,l,m$ are not all equal. In the second stage, each node $i$ transmits $\widetilde{\mathbf{x}}_{i}$ to the PS over the intermittent link $\tau_{i}$, and the PS computes the global estimate. The pseudocode for PriCER is given in Algs. 1 and 2

III Mean Squared Error Analysis

The goal of PriCER is to obtain an unbiased estimate of $\overline{\mathbf{x}}$ at the PS. Since each node sends its data to all other neighboring nodes, the PS receives multiple copies of the same data. Lemma III.1, below gives a sufficient condition to ensure unbiasedness. This is the same condition as [15, Lemma 3.1], and holds true even for PriCER.

Lemma III.1

Let the weights $\{\alpha_{ij}\}_{i,j\in[n]}$ satisfy

[TABLE]

for every $i\in[n]$. Then, $\mathbb{E}\left[{\widehat{\overline{\mathbf{x}}}\mid\{\mathbf{x}_{i}\}_{i\in[n]}}\right]=\overline{\mathbf{x}}=\frac{1}{n}\sum_{i=1}^{n}\mathbf{x}_{i}.$

We prove this lemma in App. B.

Under this unbiasedness condition, we derive a worst-case upper bound for the MSE in Thm. III.2.

Theorem III.2

Given $\mathbf{p},\mathbf{P}$ and $\mathbf{A}$ such that (3) holds, and $\mathbf{n}_{ij}\sim{\cal N}(\mathbf{0},\sigma^{2}\mathbf{I}_{d})\;\forall\;i,j\in[n]$, the MSE with PriCER satisfies,

[TABLE]

where $\sigma_{\rm tv}^{2}(\mathbf{p},\mathbf{P},\mathbf{A})$ is an upper bound on the variance induced by the stochasticity due to intermittent topology given by,

[TABLE]

and $\sigma_{\rm pr}^{2}(\mathbf{p},\mathbf{P},\sigma)$ is the variance due to the privacy given by,

[TABLE]

Thm. III.2 is derived in App. C. From (4), we see that $\sigma_{\rm pr}^{2}(\mathbf{p},\mathbf{P},\sigma)$ is the price of privacy. For a non-private setting, i.e., $\sigma=0$, the privacy induced variance $\sigma_{\rm pr}^{2}(\mathbf{p},\mathbf{P},\sigma)=0$, and Thm. III.2 simplifies to [15, Thm. 3.2]. In the following section, we introduce our privacy guarantee and the corresponding constraints leading to a choice of weight matrix $\mathbf{A}$ for the optimal Utility (MSE) - Privacy tradeoff of PriCER.

IV Privacy Analysis

PriCER yields privacy guarantees because of two reasons: $\rm(i)$ the local noise added at each node, and $\rm(ii)$ the intermittent nature of the connections. We consider the local differential privacy when any eavesdropper (possibly including the receiving node) can observe the transmission from node $i$ to node $j$ in stage-$1$ of PriCER. Let us denote the local dataset of node $i$ as ${\cal D}_{i}$. In DME, ${\cal D}_{i}$ is a singleton set and by observing the transmission from node $i$ to node $j$, the eavesdropper should not be able to differentiate between the events $\mathbf{x}_{i}\in{\cal D}_{i}$ and $\mathbf{x}_{i}^{\prime}\in{\cal D}_{i}$, where $\mathbf{x}_{i}^{\prime}\neq\mathbf{x}_{i}$. The following Thm. IV.1 (derived in App. D) formally states this guarantee.

Theorem IV.1

Given $\mathbf{n}_{ij}\sim{\cal N}(\mathbf{0},\sigma^{2}\mathbf{I}_{d})$, $\mathbf{x}_{i},\mathbf{x}_{i}^{\prime}\in\mathbb{R}^{d}$ with $\lVert\mathbf{x}_{i}\rVert_{2},\lVert\mathbf{x}_{i}^{\prime}\rVert_{2}\leq{\rm R}$, and any $\delta_{ij}\in(0,1]$, for pairs $(\epsilon_{ij},p_{ij}\delta_{ij})$ satisfying

[TABLE]

the transmitted signal from node $i$ to node $j$, $\widetilde{\mathbf{x}}_{ij}=\tau_{ij}(\alpha_{ij}\mathbf{x}_{i}+\mathbf{n}_{ij})$ is $(\epsilon_{ij},p_{ij}\delta_{ij})$-differentially private, i.e., it satisfies

[TABLE]

for any measurable set $\cal S$.

Setting $\delta:=p_{ij}\delta_{ij}$, we can immediately see that intermittent connectivity inherently boosts privacy, since for the same $\delta$ for any pair $i,j\in[n]$, the privacy level $\epsilon_{ij}$ is proportional to $\ln\left(1.25p_{ij}/\delta\right)^{\frac{1}{2}}$, implying that a smaller $p_{ij}$ leads to a stronger privacy guarantee. Additionally, from (6), the privacy guarantee $\epsilon_{ij}$ is directly related to the weight $\alpha_{ij}$. That is, if node $i$ trusts node $j$ more, $\epsilon_{ij}$ can be relatively large, and consequently, node $i$ can assign a higher weight to the data it sends to node $j$. On the other hand, if node $i$ does not trust node $j$ as much, a smaller value will be assigned to $\alpha_{ij}$. In other words, for the same noise variance $\sigma$, node $i$ will scale the signal $\alpha_{ij}$ so as to reduce the effective signal-to-noise ratio in settings where a higher privacy is required. Finally, when $p_{ij}=0$, PriCER ensures $\alpha_{ij}=0$, implying $\epsilon_{ij}=0$, i.e., perfect privacy, albeit zero utility. Our weight optimization (§V) aims to minimize the MSE subject to the privacy constraints imposed by (6).

V Privacy Constrained Weight Optimization

When deriving the utility-privacy tradeoff, our objective is to minimize the MSE at the PS subject to desired privacy guarantees, namely $(\underline{\epsilon}_{ij},\underline{\delta}_{ij}p_{ij})$ node-node differential privacy. Here, $\underline{\epsilon}_{ij},\underline{\delta}_{ij}$ are pre-designated system parameters that quantify the extent to which node $i$ trusts node $j$ (or alternatively, how much it trusts the communication link $i\to j$ against an eavesdropper). More specifically, we solve:

[TABLE]

The above optimization (V) is not necessarily convex. Furthermore, the objective is also not separable with respect to $\mathbf{A}$ and $\sigma$. Thus, in what follows, we propose an alternate minimization scheme, where we iteratively minimize with respect to $\mathbf{A}$ and $\sigma$; one variable at a time, keeping the other fixed. We tie up the components of §V-B and §V-A and present the complete PriCER weight and variance optimization algorithm in Alg. 3. For clarity of presentation, we assume that $p_{i}>0$ for all $i\in[n]$, so we can have a simple initialization rule.

V-A Optimizing variance $\sigma$ for a given weights $\mathbf{A}$

The non-negative weights, unbiasedness, and privacy constraints are present in (V) due to our problem formulation. However, when using an alternating optimization we must choose a variance that can fulfill the unbiasedness condition in the weight optimization stage. In other words, PriCER needs to add a minimum amount of noise, $\sigma_{\rm thr}$, in order to meet privacy constraints and unbiasedness conditions simultaneously. Thus, we introduce a necessary condition to ensure a non-empty feasible set when we optimize the weight for the chosen $\sigma$.

We visualize this in Fig. 2. Note that for a fixed $i\in[n]$, the unbiasedness constraint together with $\alpha_{ij}\geq 0$, defines a hyperplane ${\cal H}$ in the positive quadrant of $\mathbb{R}^{n}$ with respect to the optimization variables $\{\alpha_{ij}\}_{j\in[n]}$. Moreover, for $i\in[n]$, the constraints $\alpha_{ij}\geq 0$ and $\alpha_{ij}\leq\underline{\epsilon}_{ij}\sigma\cdot([2\ln(1.25/\underline{\delta}_{ij})]^{\frac{1}{2}}2{\rm R})^{-1}$ $\forall\;j\in[n]$, together define a box ${\cal B}$ aligned with the standard basis of $\mathbb{R}^{n}$ with one of the vertices at the origin. The edge length of this box along any of the axes is proportional to $\sigma$ When $\sigma=0$, i.e., no privacy noise is added, ${\cal B}=\mathbf{0}_{n}$, where $\mathbf{0}_{n}$ denotes the origin of $\mathbb{R}^{n}$. Since ${\cal H}$ does not pass through $\mathbf{0}_{n}$, (V-B) is infeasible. This implies there is a minimum value of $\sigma$ so that ${\cal B}$ is big enough to have a non-zero intersection with ${\cal H}$. More specifically, we require $\sigma$ such that, $\sigma\sum_{j\in[n]}p_{j}p_{ij}\underline{\epsilon}_{ij}\left(\left[2\ln\left(\frac{1.25}{\underline{\delta}_{ij}}\right)\right]^{\frac{1}{2}}2{\rm R}\right)^{-1}\geq 1,\forall\>i\in[n]$, and hence, we have the last feasibility constraint in (V-A), where

[TABLE]

We now fix $\mathbf{A}$ in (V-A) and minimize the PIV, i.e.,

[TABLE]

It can be shown (App. §E-A) that the update rule is given by:

[TABLE]

V-B Optimizing weights $\mathbf{A}$ for a given variance $\sigma$

Firstly, for a fixed $\sigma$, we minimize the weights $\mathbf{A}$, i.e.,

[TABLE]

The objective function of problem (V-B) is not convex. With this in mind, we adopt an approach similar to [15, 17] wherein (V-B) is minimized in two iterative stages – $\rm(i)$ first, a convex relaxation of (V-B) is minimized using Gauss-Seidel method, and $\rm(ii)$ the outcome is subsequently fine-tuned again, using Gauss-Seidel on (V-B). The convex relaxation is chosen to be:

[TABLE]

where the new objective function is,

[TABLE]

We delegate the complete derivations to App. §V and only mention the update rules here. Let us denote the $i^{\rm th}$ row of $\mathbf{A}$ as $\mathbf{A}_{i}$. Since the objective function of both (V-B) and (V-B) are separable with respect to $\mathbf{A}_{i}$, we can apply Gauss-Seidel iterations on both (12) and subsequently on (V-B).

Minimizing the convex relaxation (12): Let us denote the iterate at the $\ell^{\rm th}$ Gauss-Seidel iteration of the convex relaxation (V-B), as $\mathbf{A}^{(\ell)}$. Then, the update rule is given by,

[TABLE]

where $\mathds{1}_{\{\cdot\}}$ denotes the indicator function. In one iteration, only the $i^{\rm th}$ row, i.e., the weights assigned by the $i^{\rm th}$ node for its neighbors, are updated. Since Gauss-Seidel performs block-wise descent, the update $\widehat{{\mathbf{A}}}_{i}^{(\ell)}\equiv\{\widehat{\alpha}_{ij}\}_{i,j\in[n]}$ can be obtained by formulating the Lagrangian (App. §E-B). Let us denote $\widetilde{w}_{ij}\triangleq\underline{\epsilon}_{ij}\sigma([2\ln(\frac{1.25}{\underline{\delta}_{ij}})]^{\frac{1}{2}}2{\rm R})^{-1}$ and

[TABLE]

We separate the solution into three scenarios:

$\boldsymbol{p_{i}<1}$** and $\boldsymbol{p_{j}p_{ij}<1}$ for all $\boldsymbol{j\in[n]}$**

In this case,

[TABLE]

where $\widetilde{\alpha}_{ij}(\lambda_{i})$ is given by,

[TABLE]

Here, $(a)^{+}\triangleq\max\{a,0\}$, and $\lambda_{i}\geq 0$ is set such that $\sum_{j\in[n]}p_{j}p_{ij}\widehat{\alpha}_{ij}=1$. $\lambda_{i}$ is found using bisection search.

$\boldsymbol{p_{i}<1}$** and there exists $\boldsymbol{j\neq i}$ such that $\boldsymbol{p_{j}p_{ij}=1}$**

Denote $S_{i}=\sum_{k\in[n]}\mathds{1}_{\{p_{k}p_{ik}=1\}}\widetilde{w}_{ik}$. If $S_{i}\geq 1$, then we choose, $\widehat{\alpha}_{ij}=\widetilde{w}_{ij}/S_{i}$ for all $j$ such that $p_{j}p_{ij}=1$, and $\widehat{\alpha}_{ij}=0$ otherwise. If $S_{i}<1$, we set $\widehat{\alpha}_{ij}=\widetilde{w}_{ij}$ for nodes $j$ that satisfy $p_{j}p_{ij}=1$, and subsequently allocate the residual $1-S_{i}$ of the unbiasedness condition to minimize the objective function. Similar to (16), this will yield that: $\widehat{\alpha}_{ij}=\min\left\{\widetilde{\alpha}_{ij}(\lambda_{i}),\widetilde{w}_{ij}\right\}$, where $\widetilde{\alpha}_{ij}(\lambda_{i})$ is given by,

[TABLE]

Here, $\lambda_{i}\geq 0$ is such that $\sum_{j:p_{j}p_{ij}\in(0,1)}p_{j}p_{ij}\widehat{\alpha}_{ij}=1-S_{i}$.

$\boldsymbol{p_{i}=1}$

In this case, to preserve privacy we set,

[TABLE]

Fine tuning (V-B): We now fine tune the solution of above by setting it as a warm start initialization and performing Gauss-Seidel on (V-B). Then, the update equation for fine tuning is of the same form as (14) and (16)-(19). However, we plug-in the updated quantity $\widetilde{\alpha}_{ij}(\lambda_{i})$ for this case, which is now,

[TABLE]

VI Numerical Simulations

In Fig. 3, we consider a setup with $n=10$ nodes that can collaborate over an Erdős-Rényi topology, i.e., $P_{ij}={\rm p_{c}}$ for $j\neq i$ and $P_{ii}=1$. The nodes can communicate to the PS with probabilities $\mathbf{p}=[0.1,0.1,0.8,0.1,0.1,0.9,0.1,0.1,0.9,0.1]$, i.e., only three clients have good connectivity. Even though any node can communicate with any other node with a non-zero probability, they do not do so because they only trust a small number of immediate neighbors, which is varied along the $\rm x$-axis. If node $i$ trusts node $j$, we set $\epsilon_{ij}=\epsilon_{\rm high}=10^{3}$ (low privacy), otherwise, $\epsilon_{ij}=\epsilon_{\rm low}=0.1$ (high privacy). Moreover, $\epsilon_{ii}=\epsilon_{\rm high}$. We also set $\delta_{ij}=\delta=10^{-3}$. $\rm y$-axis shows the (optimized) objective value of (V), i.e., the (worst-case) upper bound to MSE. As is evident from Fig. 3, the MSE decreases as nodes trust more neighbors, as expected.

In Fig. 4, we consider that the data at each node is generated from a Gaussian distribution ${\cal N}(0,1)$, raised to the power of $3$, and normalized. This generates a heavy-tailed distribution. Consequently, if a node that has a vector with a few large coordinate values is unable to convey its data to the PS due to a failed transmission, this can incur a significant MSE. In this setup only some nodes have good connectivity to the PS, i.e., $p_{i}=p_{\rm good}=0.9$, and the remaining have $p_{i}=p_{\rm bad}=0.2$ In the naïve strategy, the PS averages whatever it successfully receives, i.e., it computes the mean estimate as $\frac{1}{n}\sum_{i\in[n]}\tau_{i}\mathbf{x}_{i}$. Whereas, in our collaborative strategy, each node trust $6$ other neighbors and can communicate with them with a probability $P_{ij}=0.8$. Clearly, PriCER achieves a lower MSE than the naïve strategy. The plots are averaged over $50$ realizations.

VII Conclusions

In this paper, we considered the problem of mean estimation over intermittently connected networks with collaborative relaying subject to peer-to-peer local differential privacy constraints. The nodes participating in the collaboration do not trust each other completely and, in order to ensure privacy, they scale and perturb their local data when sharing with others. We have proposed a two-stage consensus algorithm (PriCER), that takes into account these peer-to-peer privacy constraints to jointly optimize the scaling weights and noise variance so as to obtain an unbiased estimate of the mean at the PS that minimizes the MSE. Numerical simulations showed the improvement of our algorithm relative to a non-collaborative strategy in MSE for various network topologies. Although this work considers peer-to-peer privacy, there can be other sources of privacy leakage such as central DP at the PS too. Moreover, adding correlated privacy noise may help reduce the MSE even further. Our future work will include investigating these questions in more detail.

Appendix A A brief refresher on Differential Privacy

Definition A.1

($(\epsilon_{i},\delta_{i})$-LDP [18]) Let $\mathcal{D}_{i}$ be a set of all possible data points at node $i$. For node $i$, a randomized mechanism $\mathcal{M}_{i}:\mathcal{D}_{i}\rightarrow\mathbb{R}^{d}$ is $(\epsilon_{i},\delta_{i})$ locally differentially private (LDP) if for any $x,x^{\prime}\in\mathcal{D}_{i}$, and any measurable subset $\mathcal{S}\subseteq\text{Range}(\mathcal{M}_{i})$, we have

[TABLE]

*The setting when $\delta_{i}=0$ is referred as pure $\epsilon_{i}$-LDP.

In this work, we analyze the privacy level achieved by PriCER algorithm that adds synthetic noise perturbations to privatize its local data. We focus on analyzing the privacy leakage under an additive noise mechanism that is drawn from Gaussian distribution. This well-known perturbation technique is known as Gaussian mechanism, and it provides rigorous privacy guarantees, defined next.

Definition A.2

(Gaussian Mechanism [18]) Suppose a node wants to release a function $f(X)$ of an input $X$ subject to $(\epsilon,\delta)$-LDP. The Gaussian release mechanism is defined as:

[TABLE]

If the sensitivity of the function is bounded by $\Delta_{f}$, i.e., $\|f(x)-f(x^{\prime})\|_{2}\leq\Delta_{f}$, $\forall x,x^{\prime}$, then for any $\delta\in(0,1]$, Gaussian mechanism satisfies $(\epsilon,\delta)$-LDP, where

[TABLE]

Appendix B Proof of Lemma III.1

The proof of this is the same as [15, Lemma 3.1] while taking into account the fact that the privacy noise added is zero-mean Gaussian. In particular, note that since node $i$ sends $\tau_{ij}(\alpha_{ij}\mathbf{x}_{i}+\mathbf{n}_{ij})$ to node $j$, the total expected contribution of $\mathbf{x}_{i}$ at the PS is,

[TABLE]

where $\rm(i)$ follows since $\mathbf{n}_{ij}$ is independent of $\tau_{j}$ and $\tau_{ij}$. Substituting (3) completes the proof of the lemma.

Appendix C Proof of Theorem III.2

Note that the global estimate at the PS is:

[TABLE]

Consequently, the MSE can be written as,

[TABLE]

where the expectation is taken over the random connectivity and the local perturbation mechanism. Here, $\rm(i)$ follows because for any $j\in[n]$, the cross term is $\mathbb{E}\left[\sum_{i\in[n]}\tau_{j}\tau_{ij}\alpha_{ij}\mathbf{x}_{j}^{\top}\mathbb{E}[\mathbf{n}_{ij}]\right]=0$.

The first term TIV in (26) is solely affected by the intermittent connectivity of nodes. It can be upper bounded in exactly as is done for ColRel in [15, Thm. 3.2], and we get,

[TABLE]

To simplify PIV, which depends on the privacy noise variance, we push $\tau_{j}$ inside $\sum_{i\in[n]}$ and interchange $i,j\in[n]$ to get,

[TABLE]

Expanding the first term of (28) yields,

[TABLE]

where the second term is zero due to our assumption of uncorrelated privacy noise, i.e., $\mathbb{E}[\mathbf{n}_{ij}^{\top}\mathbf{n}_{ik}]=0$. For the same reason, expanding the second term of (28) yields,

[TABLE]

This completes the proof.

Appendix D Proof of Theorem IV.1

We begin by considering the two cases of successful transmission, i.e., $(\tau_{ij}=1)$ and unsuccessful transmission, i.e., $(\tau_{ij}=0)$ separately. Note that when $\tau_{ij}=0$, we have perfect privacy, i.e.,

[TABLE]

since there is no transmission from node $i$, i.e., $\widetilde{\mathbf{x}}_{ij}=\mathbf{0}$ irrespective of whether $\mathbf{x}_{i}\in{\cal D}_{i}$ or $\mathbf{x}_{i}^{\prime}\in{\cal D}_{i}$. When $\tau_{ij}=1$, node $j$ receives $\widetilde{\mathbf{x}}_{ij}=\alpha_{ij}\mathbf{x}_{i}+\mathbf{n}_{ij}$. The $\ell_{2}$-sensitivity is,

[TABLE]

Using (32) above, from the guarantees of Gaussian mechanism [18], when $\tau_{ij}=1$, we have for any $\delta_{ij}\in(0,1)$, the transmission from node $i$ to node $j$ is $(\epsilon_{ij},\delta_{ij})$ - differentially private, i.e.,

[TABLE]

where,

[TABLE]

Combining (31) and (33), we have,

[TABLE]

Here, $\rm(i)$ holds true since $e^{\epsilon_{ij}}\geq 1$. Note that when $p_{ij}=0$, node $i$ can never transmit to node $j$ and as a consequence, we can choose $\alpha_{ij}=0$. This completes the proof.

Appendix E Privacy-constrained weight optimization

E-A Optimizing noise variance $\sigma$ for a given $\mathbf{A}$

We start with optimizing $\sigma$ when $\mathbf{A}$ is fixed, i.e., we solve,

[TABLE]

It is easy to see that the minimizer of the optimization problem (E-A) above is $\sigma=\max\{\sigma_{a},\sigma_{\rm{thr}}\}$, where

[TABLE]

E-B Optimizing weight matrix $\mathbf{A}$ for a given $\sigma$

We now look how to optimize for $\mathbf{A}$ when the noise variance $\sigma$ is fixed. As described in §V, we first minimize a convex relaxation, and then subsequently fine tune the solution. We look at the finer details now.

Solving the convex relaxation: From [17, Lemmas $2$ and $7$], it can be seen, $\rm(i)$ $\sigma_{\rm tv}^{2}(\mathbf{p},\mathbf{P},\mathbf{A})\leq\overline{\sigma}_{\rm tv}^{2}(\mathbf{p},\mathbf{P},\mathbf{A})$, and $\rm(ii)$ $\overline{\sigma}_{\rm tv}^{2}(\mathbf{p},\mathbf{P},\mathbf{A})$ is convex in $\mathbf{A}$, showing (12) (restated below as (E-B)) is a convex relaxation of (V-B).

[TABLE]

Let $\mathbf{A}_{i}$ denotes the $i^{\rm th}$ row of $\mathbf{A}$. As the domain of (E-B) is separable with respect to $\mathbf{A}_{i}$, we use Gauss-Seidel method [25, Prop. 2.7.1] to converge to the minimizer of (E-B). Subsequently, we use the solution obtained from above as the starting point to perform Gauss-Seidel iterations again on the original problem (V-B) so as to converge to a stationary point.

Let $\mathbf{A}_{i}^{(\ell)}$ denote the iterate in the $\ell^{\rm th}$ iteration of the Gauss-Seidel method of (E-B). Here, $\mathbf{A}_{i}$ denotes the minimize of (V-B). Denote the initial iteration by $\mathbf{A}^{(0)}$. We can improve our iterates using Gauss-Seidel method until convergence to an optimal point of (V-B) as follows. At every iteration $\ell$ we compute $\mathbf{A}^{(\ell)}$ using,

[TABLE]

where,

[TABLE]

The Lagrangian of (E-B) is

[TABLE]

Evaluating the gradients,

[TABLE]

From the KKT conditions, we can optimize $\widehat{\alpha}_{ij}^{(\ell)}$ according to (15)-(19). When the Lagrange multiplier $\lambda_{i}\geq 0$ is used, we set it such that $\sum_{j\in[n]}p_{j}p_{ij}\alpha_{ij}(\lambda_{i})=1$. We can find $\lambda_{i}$ using the bisection method over the interval:

[TABLE]

Fine tuning the solution: We are now ready to fine tune the solution of the convex relaxation obtained from above. The derivation of the update equations for this is the similar to that of the convex relaxation. We skip the details and only point out the differences. The original problem (V-B) and (E-B) differ in the last term, i.e., (V-B) has $\alpha_{ij}\alpha_{ji}$ instead of $\alpha_{ij}^{2}$. Denoting the $\ell^{\rm th}$ iterate of the (fine-tuning) Gauss-Seidel iterations as $\mathbf{A}_{i}$, the update equation is the same as (38), with $\mathbf{A}_{i}^{\ell}$ instead of $\mathbf{A}_{i}^{\ell}$. Consequently, the corresponding expressions for (E-B), and the new Lagrangian have $2\sum_{j\in[n]}p_{i}p_{j}(E_{\{i,j\}}-p_{ij}p_{ji})\alpha_{ij}\alpha_{ji}$, instead of $\sum_{j\in[n]}p_{i}p_{j}(E_{\{i,j\}}-p_{ij}p_{ji})\alpha^{2}_{ij}$. Once again, using the KKT conditions, we can optimize $\widehat{\alpha}_{ij}^{(\ell)}$ according to (16)-(19), where $\widetilde{\alpha}_{ij}(\lambda_{i})$ is now given by,

[TABLE]

As before, $\lambda_{i}\geq 0$ is set such that $\sum_{j\in[n]}p_{j}p_{ij}\alpha_{ij}(\lambda_{i})=1$, and once again, we can find $\lambda_{i}$ using the bisection method over the interval:

[TABLE]

This completes the derivation in this section.