Error Bounds and Guidelines for Privacy Calibration in Differentially Private Kalman Filtering

TL;DR

This paper derives error bounds and provides guidelines for calibrating privacy levels in differentially private Kalman filtering, ensuring accurate state estimation while protecting sensitive data in control systems.

Contribution

It introduces error and entropy bounds for differentially private Kalman filtering and offers practical guidelines for privacy calibration to maintain desired accuracy.

Findings

Derived bounds on a priori and a posteriori errors.

Provided entropy bounds for privatized trajectories.

Demonstrated calibration guidelines through simulations.

Abstract

Differential privacy has emerged as a formal framework for protecting sensitive information in control systems. One key feature is that it is immune to post-processing, which means that arbitrary post-hoc computations can be performed on privatized data without weakening differential privacy. It is therefore common to filter private data streams. To characterize this setup, in this paper we present error and entropy bounds for Kalman filtering differentially private state trajectories. We consider systems in which an output trajectory is privatized in order to protect the state trajectory that produced it. We provide bounds on a priori and a posteriori error and differential entropy of a Kalman filter which is processing the privatized output trajectories. Using the error bounds we develop, we then provide guidelines to calibrate privacy levels in order to keep filter error within pre-specified bounds. Simulation results are presented to demonstrate these developments.

Equations

$$P_T\left[x\right]=\{\begin{array}{ll}x\left(k\right)& k\le T\\ 0& k>T\end{array}\text{,}$$

$${\text{Adj}}_B(v,w)=\{\begin{array}{ll}1& \Vert v-w{\Vert }_{{\ell }_2}\le B\\ 0& \text{otherwise}.\end{array}$$

$$\mathbb{P}\left[M\left(x\right)\in S\right]\le e^{ϵ}\mathbb{P}\left[M\left(x^{\prime }\right)\in S\right]+\delta \text{ for all }S\in {\Sigma }_2^q.$$

$$x(k+1)$$

$$y(k)$$

$${\Delta }_{{\ell }_2}y:=\underset{x,x^{\prime }\mid {\text{Adj}}_B(x,x^{\prime })=1}{\sup }{\Vert Cx-C{x}^{\prime }\Vert }_{{\ell }_2}.$$

$$\tilde{y}(k)=y(k)+v(k),$$

$$\sigma \ge \frac{{\Delta }_{{\ell }_2}y}{2ϵ}\left(K_{\delta }+\sqrt{K_{\delta }^2+2ϵ}\right)\text{ where }{K}_{\delta }:={\mathcal{Q}}^{-1}\left(\delta \right).$$

$$x(k+1)$$

$$y(k)$$

$$\tilde{y}\left(k\right):=y\left(k\right)+v\left(k\right)=Cx\left(k\right)+v\left(k\right),$$

$$E\big{[}\|x(k)-\hat{x}^{-}(k)\|^{2}\big{]}\text{ and }E\big{[}\|x(k)-\hat{x}(k)\|^{2}\big{]},$$

$${\hat{x}}^{-}(k+1)=H\hat{x}(k),$$

$$\hat{x}(k+1)={\hat{x}}^{-}(k+1)+\overline{\Sigma }{C}^T{V}^{-1}(\tilde{y}(k+1)-C{\hat{x}}^{-}(k+1)).$$

$$\overline{\Sigma }$$

$$=(C^T{V}^{-1}C+{\Sigma }^{-1}{)}^{-1}.$$

$$\Sigma$$

$$=H{\left({\Sigma }^{-1}+C^T{V}^{-1}C\right)}^{-1}{H}^T+W.$$

$${\lambda }_n(S)\text{tr}(K)\le \text{tr}(KS)\le {\lambda }_1(S)\text{tr}(K).$$

$${\lambda }_1(K)+{\lambda }_n(S)$$

$${\lambda }_n(K)+{\lambda }_n(S)$$

$$C^T{V}^{-1}C=\text{diag}\left(\frac{C_{11}^2}{{\sigma }_1^2},\dots ,\frac{C_{nn}^2}{{\sigma }_n^2}\right),$$

$$\{\begin{array}{l}l:=\arg {\min }_{1\le i\le n}\frac{C_{ii}^2}{{\sigma }_i^2}\\ u:=\arg {\max }_{1\le i\le n}\frac{C_{ii}^2}{{\sigma }_i^2}\end{array}$$

$$\{\begin{array}{ll}{\lambda }_n\left(C^T{V}^{-1}C\right)& =\frac{C_l^2}{{\sigma }_l^2}\\ {\lambda }_1\left(C^T{V}^{-1}C\right)& =\frac{C_u^2}{{\sigma }_u^2}.\end{array}$$

$$\text{tr}W+\frac{{\sigma }_u^2\text{tr}(H^{T}H){\lambda }_n(W)}{{\sigma }_u^2+{\lambda }_n(W)C_u^2}\le \text{tr}\Sigma \le \text{tr}W+\frac{{\sigma }_l^2\text{tr}(H^{T}H)}{C_l^2}.$$

$$\text{tr}\Sigma -\text{tr}W$$

$$\displaystyle=\text{tr}\Big{[}H^{T}H(\Sigma^{-1}+C^{T}V^{-1}C)^{-1}\Big{]},$$

$$\text{tr}\Sigma -\text{tr}W$$

$$=\frac{\text{tr}(H^{T}H)}{{\lambda }_1\left({\Sigma }^{-1}+C^T{V}^{-1}C\right)}$$

$$\displaystyle\geq\frac{\text{tr}(H^{T}H)}{\lambda_{1}(\Sigma^{-1})+\lambda_{1}\big{(}C^{T}V^{-1}C\big{)}}$$

$$\displaystyle=\frac{\text{tr}(H^{T}H)}{\frac{1}{\lambda_{n}(\Sigma)}+\lambda_{1}\big{(}C^{T}V^{-1}C\big{)}},$$

$$\text{tr}\Sigma -\text{tr}W\ge \frac{{\sigma }_u^2\text{tr}(H^{T}H){\lambda }_n(W)}{{\sigma }_u^2+{\lambda }_n(W)C_u^2},$$

$$\text{tr}\Sigma -\text{tr}W$$

$$=\frac{\text{tr}(H^{T}H)}{{\lambda }_n\left({\Sigma }^{-1}+C^T{V}^{-1}C\right)}$$

$$\le \frac{\text{tr}(H^{T}H)}{{\lambda }_n({\Sigma }^{-1})+{\lambda }_n(C^T{V}^{-1}C)}\le \frac{{\sigma }_l^2\text{tr}(H^{T}H)}{C_l^2},$$

$$\frac{n{\sigma }_u^2}{C_u^2+{\sigma }_u^2{\lambda }_n^{-1}(W)}\le \text{tr}\overline{\Sigma }\le n\frac{{\sigma }_l^2}{C_l^2}.$$

Full text

I Introduction

Emerging technologies such as smart cities [1], intelligent transportation systems [2], and smart power grids [3] all promise to improve user services with a data-driven approach. A unifying theme in these applications is the reliance on user data in driving decisions about traffic routing, power generation, and other system behaviors. Simultaneously, these data streams have been shown to be quite revealing about users, potentially disclosing their daily habits [4] and their locations [5]. Thus there has arisen a need for user privacy. Given the need for data in decision making, privacy must also preserve the usefulness of privatized data to its recipient.

In recent years, differential privacy has become a common framework for privacy of this kind. Differential privacy started in the database literature [6] and is used to privatize database entries when database queries are made. More recently, differential privacy has been extended to trajectories arising in systems and control in [7], where the goal is to preserve the privacy of whole trajectories of data as they are generated.

Differential privacy is simple to implement because it merely requires adding noise to sensitive data (or functions of sensitive data). Differential privacy has several other properties that make it useful in control system applications. In particular, it is immune to post-processing and robust to side-information, which means that its privacy guarantees are invariant under post-hoc transformations and that they are not weakened by much with the availability of auxiliary information [6]. As a result, further computations can be performed once data is privatized without harming differential privacy’s guarantees. Of course, adding noise changes the accuracy of these computations relative to their noise-free counterparts, and the effects of differential privacy have been investigated in several contexts, e.g., [8, 9, 6, 10, 7, 11].

Data-driven systems must share data to run. In combination with the vulnerability of dynamic data streams, this need has stimulated the use of differential privacy along with model-based state estimators to both protect sensitive data and make useful control decisions in the presence of privacy. Differential privacy’s immunity to post-processing means that state estimation and filtering can be performed freely without threatening the privacy guarantees of a system’s data. The Kalman filter is a widely used state estimator which has been shown to improve the utility of privatized data in various settings [12, 7]. One common approach to differentially private Kalman filtering is to have systems add noise directly to system outputs; this approach is broadly termed “input perturbation,” because individual systems add noise to the inputs of the Kalman filter. This approach has the advantage of privatizing all data before it is ever shared, eliminating the need for a trusted aggregator.

In this paper, we are interested in privacy over long time horizons, as in smart grids and other systems that will be active for a long time. We therefore consider a Kalman filter in steady state and we analyze the accuracy of filtering private data under the input perturbation paradigm. We consider a system with discrete-time dynamics, and we protect its state trajectory by adding noise to its outputs at each point in time. We quantify the effects of privacy in two ways. First, we use differential entropy to quantify the information content of the privatized output trajectories. Second, we investigate the practical effects of privacy by bounding the mean squared error of an external observer’s estimate of the system’s states.

While the computer science literature has devised methods to calibrate privacy based on analyzing static data [13], to the best of our knowledge, no such systematic study has been undertaken in the control theory literature for trajectory-valued data. We therefore explore the relationship between privacy levels, the amount of information revealed, and the accuracy of estimates based on privatized data. We do so in terms of system properties and dynamics, thereby directly linking control theoretic information with privacy of system trajectories. Based on the bounds we derive, we provide guidelines for selecting one’s privacy level based on the downstream filtering error it induces. Through doing so, we provide the ability to calibrate one’s privacy levels based on conventional control-theoretic concerns (i.e., filtering error), thereby enabling meaningful privacy calibration without requiring in-depth knowledge of differential privacy. In addition, this paper differs from [7, 14] because those papers design private filters, whereas we characterize a common Kalman filter setup rather than designing novel filtering strategies.

The rest of the paper is organized as follows. Section  II provides the necessary background for differential privacy and sets up the problem statement. In Section III, we outline the privacy implementation for our problem and briefly review Kalman filtering. Section IV presents the first main results of the paper, which are bounds on the differential entropy and MSE of state estimates based on private data. In Section V, we provide guidelines for calibrating privacy levels based on pre-specified error bounds. Next, we present numerical simulations in Section VI, and Section VII concludes the paper.

II Background on Differential Privacy and Problem Formulation

In this section we first briefly review the relevant privacy background as it pertains to private Kalman filtering specifically, and we refer the reader to [7] for a complete exposition. Then we state the problem that is the subject of the paper.

II-A Review of Differential Privacy

Differential privacy is a quantitative and statistical means of protecting data. Differential privacy makes it unlikely that an adversary or eavesdropper can make high-fidelity inferences about individuals by looking at their privatized data. It is enforced by adding noise to sensitive data (or functions thereof). Control theory provides many techniques that compensate for noise, making differential privacy a natural choice in control-theoretic settings.

Differential privacy is immune to post-processing, which means that transforming private data does not harm its privacy guarantees. In particular, filtering private trajectories does not degrade the protections of differential privacy. The guarantees of differential privacy are also robust to mechanism knowledge, which means that adversaries do not gain any advantage if they know the mechanism used to privatize data [15, 6].

In this paper, we consider a system with discrete-time dynamics. The state trajectory of the system is sensitive, and therefore it needs to be protected. Denote the system’s state trajectory by $x$. The $k^{th}$ element of $x$ is denoted by $x\left(k\right)\in\mathbb{R}^{n}$ for some $n\in\mathbb{N}$. The notion of differential privacy in this paper follows the definition of differential privacy for trajectories introduced in [7]. Differential privacy can be used to ensure that an adversary is unlikely to determine either the input or state trajectory of a system, and in this paper we implement differential privacy to protect state trajectories.

In this work, we consider the so-called “input perturbation” approach to differential privacy. This means that a system will directly add noise to its own outputs before sharing them, and this has the advantage of masking sensitive data before it is shared. Formally, a system’s state trajectory will be made approximately indistinguishable from other nearby state trajectories which that system could have produced; the notions of “nearby” and “approximately indistinguishable” are formalized below in Definitions 1 and 2.

We consider vector-valued trajectories of the general form ${Z=\left(Z\left(1\right),Z\left(2\right),\dots,Z\left(k\right),\dots\right),}$ where $Z\left(k\right)\in\mathbb{R}^{d}$ for all $k$. We also use the $\ell_{p}$-norm ${\left\|Z\right\|_{\ell_{p}}:=\left(\sum_{k=1}^{\infty}\left\|Z\left(k\right)\right\|_{p}^{p}\right)^{\frac{1}{p}},}$ where $\left\|.\right\|_{p}$ is the ordinary $p\text{-norm}$ on $\mathbb{R}^{d}$. We further define the set $\ell_{p}^{d}:=\left\{Z\mid Z\left(k\right)\in\mathbb{R}^{d},\;\left\|Z\right\|_{\ell_{p}}<\infty\right\}.$

The state trajectory $x$ is contained in the set $\tilde{\ell}_{2}^{n}$, which is the set of sequences of vectors in $\mathbb{R}^{n}$ whose finite truncations are all in $\ell_{2}^{n}$. Formally, we define the truncation operator $P_{T}$ over trajectories according to

[TABLE]

and we say that $x\in\tilde{\ell}_{2}^{n}$ if and only if $P_{T}[x]\in\ell_{2}^{n}$ for all $T\in\mathbb{N}$.

A differential privacy mechanism makes adjacent trajectories produce outputs which are similar in a precise sense. The choice of adjacency relation is a key part of any differential privacy implementation because it specifies which sensitive pieces of data must be made approximately indistinguishable. To formulate differential privacy for trajectories, we next define the adjacency relation over the space $\tilde{\ell}_{2}^{n}$ defined above.

Definition 1**.**

*(Adjacency) * Fix an adjacency parameter ${B>0}$. The adjacency relation $\textnormal{Adj}_{B}$ is defined for all $v,w\in\tilde{\ell}^{n}_{2}$ as

[TABLE]

Two state trajectories of the system are thus adjacent if the $\ell_{2}$ distance between them is not more than $B$. Differential privacy must therefore make the system’s state trajectory approximately indistinguishable from all others contained in an $\ell_{2}$-ball of radius $B$ centered on its actual trajectory.

Next is a formal definition of differential privacy for dynamical systems which specifies the probabilistic guarantees of privacy. To state it, we will use a probability space \text{(\Omega$,$\mathcal{F}$,$\mathbb{P}}). This definition considers outputs in the space $\tilde{\ell}_{2}^{q}$ and uses a $\sigma\text{-algebra}$ over $\tilde{\ell}_{2}^{q}$, denoted $\Sigma_{2}^{q}$, construction of which can be found in [7].

Definition 2**.**

($(\epsilon,\delta)$-Differential Privacy) Let* $\epsilon>0$ *and $\delta\in\left(0,\nicefrac{{1}}{{2}}\right)$ be given. A mechanism $M:\tilde{\ell}_{2}^{n}\times\Omega\rightarrow\tilde{\ell}_{2}^{q}$ is $\left(\epsilon,\delta\right)$-differentially private if for all adjacent ${x,x^{\prime}\in\tilde{\ell}_{2}^{n}}$, we have

[TABLE]

At time $k$, the system has state $x(k)\in\mathbb{R}^{n}$, with discrete-time dynamics

[TABLE]

where process noise for the system is denoted by $w\left(k\right)\in\mathbb{R}^{n}$ and the matrices $H\in\mathbb{R}^{n\times n}$ and $C\in\mathbb{R}^{q\times n}$ are time-invariant. The probability distribution of the process noise is given by $w\left(k\right)\sim\mathcal{N}\left(0,W\right)$, where $0\prec W\in\mathbb{R}^{n\times n}$, and all process noise terms are assumed to have finite variance. We assume that the matrices $H$, $W$, and $C$ are public information, representing, e.g., that the device producing outputs is of a known type.

At each time $k$, the system outputs the value $y\left(k\right)$. Absent any privacy protections, the values of $y$ could reveal those of $x$ over time, which would compromise the system’s privacy. Therefore, noise must be added to the system’s output to protect its state trajectory. That is, $y$ is what is shared, $x$ is what is sensitive, and because $y$ is a function of $x$, we add noise to $y$ to protect $x$. Calibrating the level of noise is done using the “sensitivity” of a system’s output map, which we define next for the input perturbation privacy we use; we emphasize that, although the system perturbs the outputs of its own dynamics, the “input perturbation” label applies because the system perturbs what will become the inputs to a Kalman filter. The following bound is adapted from [7, Section IV-A].

Definition 3**.**

(Sensitivity for Input Perturbation Privacy) The $\ell_{2}\text{-norm}$ sensitivity of a system’s output map is the greatest distance between two output trajectories which correspond to adjacent state trajectories. Formally, for $x,x^{\prime}\in\tilde{\ell}_{2}^{n}$,

[TABLE]

We can bound $\Delta_{\ell_{2}}y$ via $\Delta_{\ell_{2}}y\leq s_{1}(C)B$ [16], where $s_{1}(\cdot)$ is the maximum singular value of a matrix. Various mechanisms have been developed for enforcing differential privacy in the literature [6]. The Gaussian mechanism requires adding Gaussian noise to outputs to mask systems’ state trajectories, and it can be useful in control settings that are robust to Gaussian noise. We next provide a definition of the Gaussian mechanism in terms of the $\mathcal{Q}$-function, defined by $\mathcal{Q}\left(y\right)=\frac{1}{\sqrt{2\pi}}\int_{y}^{\infty}e^{-\frac{z^{2}}{2}}dz$.

Lemma 1**.**

(Input Perturbation Gaussian Mechanism) Let $\epsilon>0$ and $\delta\in\left(0,\nicefrac{{1}}{{2}}\right)$ be given. Let $y\in\tilde{\ell}_{2}^{q}$ denote the output of a system with state trajectories in $\tilde{\ell}_{2}^{n}$, and denote its ${\ell}_{2}$-norm sensitivity by $\Delta_{\ell_{2}}y$. Then the Gaussian mechanism for $\left(\epsilon,\delta\right)$-differential privacy takes the form

[TABLE]

where $v$ is a stochastic process with $v(k)\sim\mathcal{N}\left(0,\sigma^{2}I_{q}\right)$, $I_{q}$ is the $q\times q$ identity matrix, and

[TABLE]

This Gaussian mechanism provides $(\epsilon,\delta)$-differential privacy.

Proof: See [7, Corollary 1]. $\blacksquare$

In words, the Gaussian mechanism adds i.i.d Gaussian noise point-wise in time to the output of a system to keep its state trajectory private. We will use the Gaussian mechanism to enforce differential privacy for the remainder of the paper.

II-B Problem Formulation

Having covered the relevant privacy background, we now state the problem that is the focus of the paper.

Problem 1**.**

Consider a system with publicly known mean initial condition $\hat{x}^{-}(0)$, and let it have dynamics

[TABLE]

Keep the state trajectories of the system differentially private according to specified privacy parameters $(\epsilon,\delta)$. Next, investigate the effects of privacy in the following ways:

(a) Given privacy parameters $(\epsilon,\delta)$, quantify the ability of the recipients of the privatized outputs to accurately estimate the actual state trajectories of the system as a function of $\epsilon$ and $\delta$.

(b) Develop guidelines for choosing the system’s privacy parameters $(\epsilon,\delta)$ to achieve pre-specified bounds on filter error. $\diamond$

We will examine Problem 1(a) by quantifying filtering error and entropy in terms of systems’ privacy parameters. Problem 1(b) will then use these error bounds to inform how systems select their privacy parameters.

The initial state of the system is denoted by $\hat{x}^{-}(0)=\mathbb{E}[x(0)]$, where the minus sign will be used to initialize a Kalman filter which will be defined formally later. The next section presents our privacy implementation.

III Private Filtering Implementation

We consider scenarios in which systems share their privatized outputs with a recipient, such as a utility company in a smart power grid or a traffic monitor in a smart transportation system. Abstracting away implementation details, we simply say that a system sends its outputs to an aggregator, and this aggregator will run a Kalman filter. To protect its own privacy, the system only shares its privatized outputs with the aggregator. The privatized outputs of the system may also be received by other entities e.g., other systems in a network, adversaries, an eavesdropper, and data analysts. Our results apply to these other recipients as well.

Without privacy, this transmission of data could reveal the system’s state trajectory, and, as a result, compromise the system’s privacy. Hence, the system adds privacy noise at each time $k$ to its output before sharing it, giving

[TABLE]

where the privacy noise $v\left(k\right)\sim\mathcal{N}\left(0,\sigma^{2}I_{q}\right)$ as in Lemma 1. Introducing privacy naturally involves sacrificing a level of accuracy in the shared data, and the trade-offs and effects of privacy need to be rigorously evaluated to quantify the performance of private filtering.

The aggregator receives the privatized outputs of the system and implements a Kalman filter. The Kalman filter minimizes the mean squared error (MSE) of both prediction and estimation for the systems studied in this paper, which are linear systems with Gaussian noise. Mathematically, the Kalman filter minimizes both

[TABLE]

where $\hat{x}^{-}(k)$ and $\hat{x}(k)$ respectively denote the a priori state prediction and a posteriori state estimate of the Kalman filter at time $k$. As noted in Problem 1, the term $\hat{x}^{-}(0)$ is assumed to be publicly known. We consider a steady-state Kalman filter because we are interested in systems over long time horizons.

The update equation for the prediction step of the Kalman filter is evaluated as [17]

[TABLE]

and the a posteriori state estimate $\hat{x}(k)$ is updated as

[TABLE]

Assuming the observability of the pair $(H,C)$ and controllability of the pair $(A,D)$, where $W=DD^{T}$, the a posteriori error covariance matrix $\overline{\Sigma}$ is computed as

[TABLE]

Here, the a priori error covariance matrix $\Sigma$ is the unique positive semidefinite solution to the discrete algebraic Riccati equation

[TABLE]

Under differential privacy, it is provably unlikely for the recipients of $\tilde{y}(k)$ to distinguish a system’s actual state trajectory from an adjacent one. In this setting, the Kalman filter minimizes the error in state prediction and estimation in the mean square sense, which means it provides the optimal estimate of a private state trajectory. Therefore, studying the connection between the Kalman filter and data privacy can elucidate fundamental limits of information accuracy when dealing with private trajectories.

IV Quantifying the Effects of Privacy

In this section, we explicitly quantify the ability of the aggregator or any potential recipient of private data, e.g., an adversary or an eavesdropper, to uncover the state trajectory of a system using its privatized outputs. One natural way to do so is to bound the MSE of the prediction and estimation steps of a Kalman filter that processes private data. Bounding these errors as functions of the system’s privacy parameters will directly connect the system’s privacy levels to the accuracy with which the aggregator can estimate its state values. We proceed by developing trace bounds for the a priori error covariance matrix $\Sigma$ and the a posteriori error covariance matrix $\overline{\Sigma}$, which are equal to the MSE of the prediction and MSE of the estimate in the Kalman filter, respectively. Because the Kalman filter in steady state minimizes both of these quantities, lower bounds on them are lower bounds on MSE for any filtering strategy across long time horizons.

Toward doing so, the following lemma upper and lower bounds the trace of a matrix product. In it, we use $\lambda_{n}(K)\leq\dots\leq\lambda_{1}(K)$ to denote the eigenvalues of the matrix $K$.

Lemma 2**.**

Let $K$ and $S$ be $n\times n$ matrices. If $K=K^{T}\geq 0$ and $S$ is symmetric, then

[TABLE]

Proof:

See [18, Fact 5.12.4]. $\blacksquare$ ∎

We next have an analogous lemma for matrix sums.

Lemma 3**.**

Let $K$ and $S$ be $n\times n$ Hermitian matrices. Then

[TABLE]

Proof:

See [18, Theorem 8.4.11.]. $\blacksquare$ ∎

To ease the presentation of the forthcoming results, we impose the following assumption.

Assumption 1**.**

The matrix $C$ is diagonal.

Below, we will repeatedly encounter the term $C^{T}V^{-1}C$, and we present bounds that we will use below. First observe that

[TABLE]

and we define

[TABLE]

and then

[TABLE]

Next, we present lower and upper bounds for the a priori error covariance of the Kalman filter as functions of system’s privacy noise.

Theorem 1**.**

The steady state a priori error covariance of the Kalman filter, equal to $\textnormal{tr}\Sigma$, is bounded via

[TABLE]

Proof:

The steady state MSE of the predictions of the Kalman filter is equal to the trace of the a priori error covariance of the Kalman filter as given in Equation (III). Taking the trace of Equation (III), we obtain

[TABLE]

where we have used the cyclic permutation property of the trace. Next, we use Lemma 2 to write

[TABLE]

where we apply Lemma 3 on the third line to split up the eigenvalues and use the fact that $\lambda_{1}(\Sigma^{-1})=\nicefrac{{1}}{{\lambda_{n}(\Sigma)}}$ in the final step. It is shown in [19, Theorem 3.1] that $\Sigma\geq W$, and therefore $\lambda_{n}(\Sigma)\geq\lambda_{n}(W)$. Using this fact and Equation (19), we find

[TABLE]

which completes the first part of the proof. Similarly, by applying Lemmas 2 and 3 consecutively to Equation (III), $\text{tr}\Sigma$ can be upper-bounded as

[TABLE]

where in the second step we have used $\lambda_{1}(M^{-1})=\nicefrac{{1}}{{\lambda_{n}(M)}}$ and the third step uses Lemma 3 to split the eigenvalues. This completes the proof. $\blacksquare$ ∎

Theorem 1 bounds the MSE of the aggregator’s prediction of a system, which quantifies the ability of the aggregator to infer future states of the system. The following theorem presents similar bounds for $\overline{\Sigma}$, which represent the aggregator’s ability to determine the system’s current state.

Theorem 2**.**

Suppose a system shares its privatized output trajectory, and the aggregator has all public information. Then, the steady-state MSE of the a posteriori estimate of the system’s states is bounded by

[TABLE]

Proof:

The steady-state mean-squared estimation error E\big{[}\|x(k)-\hat{x}(k)\|^{2}\big{]} is equivalent to the trace of the a posteriori error covariance matrix $\overline{\Sigma}$ in Equation (III). Using Lemma 2, a lower bound for the trace of $\overline{\Sigma}$ can derived as

[TABLE]

where in the second line we have used Lemma 3 to split the eigenvalues. In the second-to-last line, we use $\Sigma\geq W$ based on [19, Theorem 3.1] to use $\lambda_{n}(\Sigma)\geq\lambda_{n}(W)$. Similarly, using Lemma 2, an upper bound for the trace of $\overline{\Sigma}$ can be derived as

[TABLE]

where in the last line $\lambda_{n}(\Sigma^{-1})>0$ is eliminated. $\blacksquare$ ∎

Together, the upper and lower bounds on $\text{tr}(\overline{\Sigma})$ give MSE bounds which elucidate the balance between privacy and accuracy of information shared with the aggregator.

Privacy and utility can be inherently conflicting goals, in the sense that the greater the level of privacy is, the less useful information will generally be. To study this relationship, we use an information theoretic tool to investigate the effects of the privacy noise $v(k)$. In particular, we consider the differential entropy in the a posteriori estimates $\hat{x}(k)$ and a priori predictions $\hat{x}^{-}(k)$, which were defined in Section III. Shannon entropy has been used to investigate the leakage of information while using differential privacy in other settings, for example in [8] and in distributed linear control systems [20]. Differential entropy is useful for Gaussian distributions because it bounds the sub-level sets of $\mathcal{R}^{-1}$, where $\mathcal{R}\left(y\right)=1-2\mathcal{Q}\left(y\right)$, which is the volume of a covariance ellipsoid. Therefore, we will quantify the effects of privacy noise upon the aggregator by studying how privacy noise affects $\ln\det\Sigma$ and $\ln\det\overline{\Sigma}$, which are within an additive and multiplicative factor of the differential entropy of error in $\hat{x}$ and $\hat{x}^{-}$, respectively. Next, we present log-determinant bounds for the a priori error covariance of the Kalman filter.

Theorem 3**.**

Suppose that

[TABLE]

where $\Gamma=\text{diag}\left(\gamma_{1},\dots,\gamma_{n}\right)$, $\gamma_{i}:=\frac{\sigma_{i}^{2}W_{ii}}{\sigma_{i}^{2}+C_{ii}^{2}W_{ii}},$ and ${s_{1}(\cdot)\geq\dots\geq s_{n}(\cdot)}$ denote the singular values of a matrix. The log-determinant of the a priori error covariance of the Kalman filter can be upper-bounded as

[TABLE]

where $\eta:=s_{n}^{2}\left(H\right)\lambda_{1}\left(\Gamma\right)+\lambda_{n}\left(W\right).$ Furthermore, the log-determinant of the a priori error covariance of the Kalman filter can be lower-bounded as

[TABLE]

Proof:

See [16]. $\blacksquare$ ∎

Next, we derive bounds on the log-determinant of the a posteriori error covariance of the Kalman filter. To facilitate the following analysis let us define the function

[TABLE]

where $X=X^{T}\succeq 0$ is a variable and the matrices $C$ and $V$ were defined in Section II. We state the following elementary lemmas that we will use below.

Lemma 4**.**

If $A\succ 0$ and $B\succeq 0,$ then there exists $t\geq 0$ such that $tA\geq B$.

Proof:

See [21, Lemma 3]. $\blacksquare$ ∎

Lemma 5**.**

For every $t\geq 0,$ we have $f(tC^{-1}VC^{-T})\preceq C^{-1}VC^{-T}$.

Proof: Inspired by the work in [21], let $t\geq 0$. Then,

[TABLE]

Lemma 6**.**

Let $S$ be an $n\times n$ Hermitian matrix. Then,

[TABLE]

Proof:

See [18, Corollary 8.4.2]. $\blacksquare$ ∎

We now present our log-determinant bounds for $\overline{\Sigma}$.

Theorem 4**.**

The log-determinant of the a posteriori error covariance of the Kalman filter is bounded via

[TABLE]

Proof:

Computing $f(\Sigma)$ in Equation (35), we get

[TABLE]

By Lemma 4, there exists a $t\geq 0$ such that $\Sigma\preceq tC^{-1}VC^{-T}$ because $\Sigma\succeq 0$ by definition and $C^{-1}VC^{-T}\succ 0$. Since $f$ is a monotonic function [21], we have $f(\Sigma)\preceq f(tC^{-1}VC^{-T})$, and therefore by Lemma 2,

[TABLE]

Taking the log-determinant of both sides, we find

[TABLE]

Next, using Lemma 6, Equation (III) implies that

[TABLE]

where, due to the similarity of the steps of this proof to the proof for Theorem 2, we have omitted the explanations for each step. Using Equation (19) and taking the log-determinant of the both sides of the Equation (IV), we can write

[TABLE]

and the theorem follows. $\blacksquare$ ∎

Of course, beyond merely studying the impacts of privacy, one can leverage these bounds to enable better privacy parameter selection by tailoring privacy levels to attain a certain quality of information downstream. That is the subject of the next section.

V Guidelines for Selecting Privacy Parameters

In this section, we develop new guidelines for selecting privacy parameters, which will allow us to achieve specified filtering error bounds. These bounds enable the calibration of privacy levels based on the desired accuracy of those making decisions with private data, as well as individuals’ privacy desires. The value of the privacy parameter $\delta$ is typically chosen to be small and fixed. The value of $\delta$ can be understood as the probability of differential privacy failing to protect sensitive data, and it is therefore often [7] chosen in $[10^{-5},0.1]$ and we adopt this for range for the rest of the paper.

Theorem 5**.**

Suppose a system shares its privatized output trajectory and the aggregator has access to all public information. Take ${\delta\in[10^{-5},10^{-1}]}$ and choose $\sigma=\frac{\Delta_{\ell_{2}}y}{2\epsilon}\left(K_{\delta}+\sqrt{K_{\delta}^{2}+2\epsilon}\right)$. Suppose we want the steady state MSE of predictions of the system’s next states, i.e., the a priori state predictions, to be in $[B_{l},B_{u}]$ for some bounds $B_{l}$ and $B_{u}$. A sufficient condition to do so is to bound $\epsilon$ via

[TABLE]

where

[TABLE]

and

[TABLE]

Proof:

First, choose $\epsilon\geq\frac{1}{8}\left(\frac{1+\sqrt{36\eta_{3}+1}}{\eta_{3}}\right)^{2}$ and solve for $\eta_{3}$ to get $\frac{9+\sqrt{2\epsilon}}{2\epsilon}\leq\eta_{3}$. Taking $\delta\in[10^{-5},10^{-1}]$ implies ${K_{\delta}\in[1,4.5]}$. As a result, we can write $\nicefrac{{(2K_{\delta}+\sqrt{2\epsilon})}}{{2\epsilon}}\leq\nicefrac{{(9+\sqrt{2\epsilon})}}{{2\epsilon}}$ to get

[TABLE]

Using the fact that $\sqrt{K+S}\leq\sqrt{K}+\sqrt{S}$, substituting for $\eta_{3}$, squaring both sides and rearranging, we have

[TABLE]

which implies that $\sigma_{l}^{2}\leq\frac{(B_{u}-\text{tr}W)C_{l}^{2}}{\textnormal{tr}(H^{T}H)}$. It then follows that

[TABLE]

which, by comparing to Theorem 1, implies that $\textnormal{tr}\Sigma\leq B_{u}$.

Next, choose $\epsilon\leq\nicefrac{{1}}{{\eta_{1}}}$. Given $K_{\delta}\geq 1$, we can write ${\epsilon\leq\nicefrac{{K_{\delta}}}{{\eta_{1}}}}$, and, rearranging the terms, we find $\eta_{1}\leq\frac{K_{\delta}}{\epsilon}.$ Substitute for $\eta_{1}$, square, and rearrange to get

[TABLE]

Now, $\frac{K_{\delta}}{\epsilon}\leq\frac{K_{\delta}+\sqrt{K_{\delta}^{2}+2\epsilon}}{2\epsilon}$ and therefore

$\frac{\left(B_{l}-\textnormal{tr}W\right)\lambda_{n}\left(W\right)C_{u}^{2}}{\textnormal{tr}(H^{T}H)\lambda_{n}\left(W\right)-B_{l}+\textnormal{tr}W}\leq\left(\Delta_{\ell_{2}}y\right)^{2}\left(\frac{K_{\delta}+\sqrt{K_{\delta}^{2}+2\epsilon}}{2\epsilon}\right)^{2}$,

which implies

[TABLE]

Therefore,

[TABLE]

and by Theorem 1, choosing $\epsilon$ as above is sufficient to guarantee $\textnormal{tr}\Sigma\geq B_{l}$. $\blacksquare$ ∎

Theorem 5 presents upper and lower bounds for the privacy parameter $\epsilon$ which ensure that steady-state a priori filtering error remains within acceptable bounds. Next, we provide analogous bounds on $\epsilon$ for a posteriori error.

Theorem 6**.**

Suppose a system shares its privatized output trajectory and the aggregator has access to all public information. Take $\delta\in[10^{-5},10^{-1}]$ and set ${\sigma=\frac{\Delta_{\ell_{2}}y}{2\epsilon}\left(K_{\delta}+\sqrt{K_{\delta}^{2}+2\epsilon}\right)}$. Suppose we want the steady-state MSE of the estimated states of the system, i.e., the a posteriori state estimates, to be contained in the interval $[B_{l},B_{u}]$ for some bounds $B_{l}$ and $B_{u}$. To do so, it is sufficient to choose the privacy parameter $\epsilon$ according to

[TABLE]

where

[TABLE]

Proof:

Choose $\epsilon\geq\frac{1}{8}\left(\frac{1+\sqrt{36\eta_{4}+1}}{\eta_{4}}\right)^{2}$ and solve for $\eta_{4}$ to get $\frac{9+\sqrt{2\epsilon}}{2\epsilon}\leq\eta_{4}$. Choosing $\delta\in[10^{-5},10^{-1}]$ gives ${K_{\delta}\in[1,4.5]}$. As a result, similar to the proof of Theorem 5, we can write $\nicefrac{{(2K_{\delta}+\sqrt{2\epsilon})}}{{2\epsilon}}\leq\nicefrac{{(9+\sqrt{2\epsilon})}}{{2\epsilon}}$ to get

[TABLE]

Because $\sqrt{K+S}\leq\sqrt{K}+\sqrt{S}$, we can lower-bound the left-hand-side to write

[TABLE]

Squaring, substituting in $\eta_{4}$, and rearranging we get

[TABLE]

which is equivalent to $\sigma_{l}^{2}\leq\frac{B_{u}C_{l}^{2}}{n},$ which implies $\frac{n\sigma_{l}^{2}}{C_{l}^{2}}\leq B_{u}.$ Comparing this result to Theorem 2, we see that this is sufficient for $\textnormal{tr}\overline{\Sigma}\leq B_{u}$.

Next, choose $\epsilon\leq\frac{1}{\eta_{2}}$. Given $K_{\delta}\in[1,4.5]$, we may write $\eta_{2}\leq\frac{K_{\delta}}{\epsilon}.$ We substitute for $\eta_{2}$ and square both sides to write

[TABLE]

and therefore, by upper-bounding the right-hand-side and rearranging we write

[TABLE]

This in turn implies $\frac{B_{l}C_{u}^{2}}{n-B_{l}\lambda_{n}^{-1}(W)}\leq\sigma_{u}^{2}.$ Isolating $B_{l}$ gives $\frac{\sigma_{u}^{2}n}{C_{u}^{2}+\sigma_{u}^{2}\lambda_{n}^{-1}(W)}\geq B_{l},$ which, in light of Theorem 2, is a sufficient condition to get $\textnormal{tr}\overline{\Sigma}\geq B_{l}$. $\blacksquare$ ∎

Theorem 6 provides guidelines for choosing the privacy parameters $(\epsilon,\delta)$, which allows a user to make informed decisions for its level of privacy. We next demonstrate these bounds in practice.

VI Case Study

In this section, we simulate a system with state $x(k)\in\mathbb{R}^{2}$ for all $k$ and dynamics

[TABLE]

We proceed to enforce input perturbation differential privacy as discussed in Section III. We choose ${(\epsilon,\delta)=(\ln 3,0.001)}$, which gives $\sigma=2.96$. The privacy noise $v\left(k\right)\sim\mathcal{N}\left(0,\sigma^{2}I_{2\times 2}\right)$ is added to the outputs $y(k)$ at each time $k$. The aggregator receives the private outputs pointwise in time and runs a Kalman filter, and we simulate this setup for $100$ timesteps.

The results of this simulation are presented in Figures 1 and 2. In Figure 1, we present the MSE bounds derived in Theorem 1, and we compare them with the actual instantaneous a priori error. On average, the a priori error in predictions of the system’s states remains within the given bounds; ephemeral bound violations are expected as these bounds pertain to mean-square error. In Figure 2, we demonstrate the instantaneous error of the estimated states of the system and we compare that with the upper and lower bounds derived in Theorem 2. As expected, the instantaneous a posteriori error typically lies within the bounds derived in Theorem 2. Both plots illustrate our bounds on the ability of an aggregator to predict or estimate the states of a system sharing privatized information.

VII Conclusions

In this paper, we have proposed new guidelines for calibrating the levels of privacy when enforcing differential privacy in linear systems with Gaussian noise. These guidelines were chosen to attain desired filtering error bounds, and novel bounds were presented for both filter entropy and filter error in terms of a system’s privacy levels. Future work includes investigating general filtering techniques in which nonlinear systems are considered, with potential applications in smart power grids and autonomous systems.