Private Inner Product Retrieval for Distributed Machine Learning

TL;DR

This paper introduces a private inner product retrieval scheme for distributed machine learning, enabling users to obtain specific inner products from servers with minimal communication while preserving data privacy.

Contribution

It formulates the problem of private inner product retrieval, develops algorithms based on multi-message PIR, and establishes convergence properties and bounds for large data files.

Findings

Achieves minimal communication load for inner product retrieval

Proves convergence of inner product distributions to uniform variables

Develops a tight converse bound for large data files

Abstract

In this paper, we argue that in many basic algorithms for machine learning, including support vector machine (SVM) for classification, principal component analysis (PCA) for dimensionality reduction, and regression for dependency estimation, we need the inner products of the data samples, rather than the data samples themselves. Motivated by the above observation, we introduce the problem of private inner product retrieval for distributed machine learning, where we have a system including a database of some files, duplicated across some non-colluding servers. A user intends to retrieve a subset of specific size of the inner products of the data files with minimum communication load, without revealing any information about the identity of the requested subset. For achievability, we use the algorithms for multi-message private information retrieval. For converse, we establish that as the length of the files becomes large, the set of all inner products converges to independent random variables with uniform distribution, and derive the rate of convergence. To prove that, we construct special dependencies among sequences of the sets of all inner products with different length, which forms a time-homogeneous irreducible Markov chain, without affecting the marginal distribution. We show that this Markov chain has a uniform distribution as its unique stationary distribution, with rate of convergence dominated by the second largest eigenvalue of the transition probability matrix. This allows us to develop a converse, which converges to a tight bound in some cases, as the size of the files becomes large. While this converse is based on the one in multi-message private information retrieval, due to the nature of retrieving inner products instead of data itself some changes are made to reach the desired result.

Equations

$$R_{\mathcal{D}}(h)=\underset{(\mathbf{x},y)\sim \mathcal{D}}{\mathrm{Pr}}\{h(\mathbf{x})\ne y\}.$$

$$\mathcal{H}=\{\mathbf{x}\mapsto \mathrm{sign}(⟨\mathbf{w},\mathbf{x}⟩+b)|\mathbf{w}\in {\mathbb{R}}^L,b\in \mathbb{R}\}.$$

$$\underset{\mathbf{w},b}{\min }\frac{1}{2}||{\mathbf{w}}^2||$$

$$y_i(⟨\mathbf{w},{\mathbf{x}}_i⟩+b)\ge 1,\forall i\in [1:m],$$

$$\underset{{\alpha }_i,i\in [1:m]}{\max }\sum _{i=1}^m{\alpha }_i-\frac{1}{2}\sum _{i,j=1}^m{\alpha }_i{\alpha }_j{y}_i{y}_j⟨{\mathbf{x}}_i,{\mathbf{x}}_j⟩$$

$$\text{subject to: }{\alpha }_i\ge 0\text{ and }\sum _{i=1}^m{\alpha }_i{y}_i=0,\forall i\in [1:m].$$

$$\mathbf{w}$$

$$\mathcal{H}=\{\mathbf{x}\mapsto ⟨\mathbf{w},\mathbf{x}⟩+b|\mathbf{w}\in {\mathbb{R}}^L,b\in \mathbb{R}\}.$$

$$\underset{\mathbf{w},b}{\min }\frac{1}{m}\sum _{i=1}^m(⟨\mathbf{w},{\mathbf{x}}_i⟩+b-y_i{)}^2,$$

$$\underset{\tilde{\mathbf{w}}}{\min }F(\tilde{\mathbf{w}})=\frac{1}{m}||{\mathbf{X}}^{\top }\tilde{\mathbf{w}}-\mathbf{y}|{|}^2,$$

$$\frac{2}{m}\mathbf{X}({\mathbf{X}}^{\top }\tilde{\mathbf{w}}-\mathbf{y})=0\iff \mathbf{X}{\mathbf{X}}^{\top }\tilde{\mathbf{w}}=\mathbf{Xy}.$$

$$\tilde{\mathbf{w}}=\{\begin{array}{ll}(\mathbf{X}{\mathbf{X}}^{\top }{)}^{-1}\mathbf{Xy}& \text{if }\mathbf{X}{\mathbf{X}}^{\top }\text{ is invertible}\\ (\mathbf{X}{\mathbf{X}}^{\top }{)}^{†}\mathbf{Xy}& \text{otherwise}\end{array}.$$

$$\tilde{\mathbf{w}}=\{\begin{array}{ll}\mathbf{X}({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}\mathbf{y}& \text{if }{\mathbf{X}}^{\top }\mathbf{X}\text{ is invertible}\\ \mathbf{X}({\mathbf{X}}^{\top }\mathbf{X}{)}^{†}\mathbf{y}& \text{otherwise}\end{array}.$$

$$\arg \underset{\mathbf{W},\mathbf{U}}{\min }\sum _{i=1}^m||{\mathbf{x}}_i-\mathbf{UW}{\mathbf{x}}_i|{|}^2.$$

$$\arg \underset{\mathbf{U}}{\min }\sum _{i=1}^m||{\mathbf{x}}_i$$

$${\mathbf{U}}^{\top }\mathbf{U}=\mathbf{I},$$

$$\mathbf{X}{\mathbf{X}}^{\top }\mathbf{Xu}=\lambda \mathbf{Xu}\Rightarrow \mathbf{AXu}=\lambda \mathbf{Xu}.$$

$$H(W_1,W_2,\dots ,W_K)=LK\log (q).$$

$$W_k=(w_{k1},...,w_{kL}{)}^{\top }$$

$$w_{k\ell }\in \mathbb{F}(q),\text{for}k\in [1:K],\ell \in [1:L].$$

$${\mathcal{X}}^{(L)}=\{⟨W_i,W_j⟩,\forall i,j\in [1:K]\}.$$

$$\mathcal{T}=\{\{i,j\},\forall i,j\in \{1,2,...,K\}\}.$$

$${\mathcal{X}}_{\mathcal{P}}^{(L)}=\{⟨W_i,W_j⟩,\forall \{i,j\}\in \mathcal{P}\}.$$

$$I(W_1,...,W_K;Q_1^{[\mathcal{P}]},...,Q_N^{[\mathcal{P}]})=0.$$

$$H(A_n^{[\mathcal{P}]}|W_1,...,W_K,Q_n^{[\mathcal{P}]})=0.$$

$$H({\mathcal{X}}_{\mathcal{P}}^{(L)}|A_{1:N}^{[\mathcal{P}]},Q_{1:N}^{[\mathcal{P}]})=0.$$

$$(Q_n^{[{\mathcal{P}}_1]},A_n^{[{\mathcal{P}}_1]},W_1,...,W_K)\sim (Q_n^{[{\mathcal{P}}_2]},A_n^{[{\mathcal{P}}_2]},W_1,...,W_K).$$

$$R(P,L)=\underset{\mathcal{P}\subseteq \mathcal{T},|\mathcal{P}|=P}{\min }\frac{H({\mathcal{X}}_{\mathcal{P}}^{(L)})}{{\sum }_{n=1}^{N}H(A_n^{[\mathcal{P}]})}.$$

$${\underline{R}}_{\mathsf{MPIR}}(K,P,N)\le C_{\mathsf{MPIR}}\le {\overline{R}}_{\mathsf{MPIR}}(K,P,N),$$

$$\frac{1}{{\overline{R}}_{\mathsf{MPIR}}(K,P,N)}=\frac{1}{{\underline{R}}_{\mathsf{MPIR}}(K,P,N)}=1+\frac{K-P}{PN},$$

$$\frac{1}{{\overline{R}}_{\mathsf{MPIR}}(K,P,N)}=\sum _{i=0}^{\text{⌊PK⌋−1}}\frac{1}{N^i}+\left(\frac{K}{P}-\lfloor \frac{K}{P}\rfloor \right)\frac{1}{N^{\text{⌊PK⌋}}},$$

$$\frac{{\sum }_{i=1}^P{\beta }_i{r}_i^{\text{K−P}}\left[{\left(1+\frac{1}{r_i}\right)}^{\text{K}}-{\left(1+\frac{1}{r_i}\right)}^{\text{K−P}}\right]}{{\sum }_{i=1}^P{\beta }_i{r}_i^{\text{K−P}}\left[{\left(1+\frac{1}{r_i}\right)}^{\text{K}}-1\right]},$$

Full text

Motivated by the above observation, we introduce the problem of private inner product retrieval for distributed machine learning, where we have a system including a database of some files, duplicated across some non-colluding servers. A user intends to retrieve a subset of specific size of the inner products of the data files with minimum communication load, without revealing any information about the identity of the requested subset. For achievability, we use the algorithms for multi-message private information retrieval. For converse, we establish that as the length of the files becomes large, the set of all inner products converges to independent random variables with uniform distribution, and derive the rate of convergence. To prove that, we construct special dependencies among sequences of the sets of all inner products with different length, which forms a time-homogeneous irreducible Markov chain, without affecting the marginal distribution. We show that this Markov chain has a uniform distribution as its unique stationary distribution, with rate of convergence dominated by the second largest eigenvalue of the transition probability matrix. This allows us to develop a converse, which converges to a tight bound in some cases, as the size of the files becomes large. While this converse is based on the one in multi-message private information retrieval due to the nature of retrieving inner products instead of data itself some changes are made to reach the desired result.

I Introduction

With the growth in data volume over recent years, the tasks of data storage and processing are often offloaded from in-house trusted systems to some external entities. Such distributed environments raise challenges, not experienced before. One of the most important ones is privacy concern, which can have different interpretations. Based on the applications use-case, the private asset might be the training data, test data, and even the model parameters (the learning algorithm). While the first two have been the subject of extensive research, from both computational cryptography and information-theoretic perspectives, the last one has been less understood.

In the privacy of the machine learning algorithms, the goal is to ensure the privacy of the parameters. Many different scenarios can be considered in which the parameters are in danger of breaching, and need to be addressed. Here, we focus on the case, where the learner must download some data samples from the servers to train the model. In this case, the learner wants to keep the identity of this subset hidden from the servers. The reason is that in many cases, revealing the identity of the selected training samples would reveal considerable information about the intention of the learner, and can be used to guess the learning algorithm and calculate parameters of the model. For example, assume that learner downloads some training samples from a server to train a classification algorithm, say support vector machines (SVM). The server can easily guess that, and run the same algorithm, and gain full knowledge about the intention and the model.

In this paper, we investigate the above privacy concern in a distributed setting, while our goal is to achieve privacy in a fundamental and information-theoretic level where no information is revealed about the algorithms to data owners. We argue that some of the most basic machine learning algorithms in different areas, including but not limited to SVM for classification, regression for relationship estimation, and principal component analysis (PCA) for dimensionality reduction, share an important feature in using sample data in their algorithm. To run these methods, the learner needs the inner products of the data files instead of the raw data. This can be particularly important when the length of input vectors is large compared to the number of data used for learning.

On a separate line of research, the privacy in distributed settings, referred to as private information retrieval (PIR), is investigated. In [1], the basic setup of PIR is studied, where the goal is to retrieve a file from a dataset, replicated in some non-colluding servers, without revealing its index. In particular, the capacity, as the infimum of the normalized download rate, is characterized. This is followed by [2, 3, 4, 5, 6] for different cases such as symmetric privacy, possibility of collusion among the servers, and coded storage instead of uncoded replication of data files in servers. In particular, in [7], the multi-message PIR (MPIR) problem is studied, where the objective is to privately download a subset of files, instead of just one, and the capacity is approximately, and in some cases tightly, characterized. The problem of retrieving a linear function of files from the servers, referred to as private computation (PC) or private function retrieval (PFR), is investigated in [8] and [9]. In [10] the capacity for private linear computation in MDS coded databases is studied. Recently the new problem of retrieving a polynomial function of files from some servers has been introduced and discussed in [11] and [12] by using Lagrange encoding in coded databases.

In this paper, we study a system, including a dataset of $K$ files, replicated across $N$ non-colluding servers. A user (learner) wishes to retrieve a subset of inner products out of all possible inner products of $K$ data files, without revealing the identity of the subset to each server. We prove that as the length of files, $L$, goes to infinity, the set of inner products of all data files (listed in the vector $X^{(L)}$) converges, in distribution, to a set of mutually independent uniform random variables. To show that, we introduce some dependencies in the sequence of $X^{(L)}$, $L=1,2,\ldots$, while keeping the marginal distribution of $X^{(L)}$ the same. Thanks to this dependency, we show that $\{X^{(L)}\}_{L=1}^{\infty}$ forms a time-homogeneous irreducible Markov chain, with uniform distribution as its unique stationary distribution. Moreover, the rate of convergence is governed by the second largest eigenvalue $\lambda_{2}$ of the transition probability matrix, where $|\lambda_{2}|\leq 1$. This property motivates us to suggest MPIR as an achievable scheme. In addition, we rely on the above property to develop a converse which becomes tight in some case, as the length of files goes to infinity. While this converse is based on [7], a few changes are needed to be made to reach our goal. This is because of the difference in retrieving inner products instead of data files in [7]. For example, the number of possible inner products cannot be any arbitrary integer which forces us to introduce an equivalent problem with arbitrary number of inner products in the process of reaching converse results.

The organization of the paper is as follows: In Section II, we discuss and motivate why retrieving the set of inner products are critical in machine learning. Next in Section III, we formally define the problem setting. We state our main results in Section V and their proofs in Section VI.

II Background and Motivation

In what follows, we review some of the most basic machine learning algorithms, in three areas of classification, regression, and dimension reduction, and show that all three are based on the inner products of the samples, rather than the samples.

1. Support vector machines (SVM): The SVM is one of the basic classification algorithms, where the goal is to correctly label the data files. This algorithm has many use cases such as face detection, bioinformatics (gene classifications), text categorization and etc. Here, we describe a simple case of SVM from [13, Page 63] and we discuss that knowing the inner products is enough to run the algorithm (instead of knowing the entire database).

Consider an input alphabet $\mathcal{X}$ consisting of length $L$ vectors, a target output alphabet $\mathcal{Y}=\{-1,1\}$ and a distribution $\mathcal{D}$ on $\mathcal{X}\times\mathcal{Y}$. The learner has $m$ training samples from $\mathcal{X}\times\mathcal{Y}$, denoted by $(\mathbf{x}_{1},y_{1}),(\mathbf{x}_{2},y_{2}),\ldots,(\mathbf{x}_{m},y_{m})$, drawn from $\mathcal{D}$. The goal here is to find function $h:\mathcal{X}\to\mathcal{Y}$ from hypothesis set $\mathcal{H}$, such that the following generalization error is minimized over $\mathcal{H}$:

[TABLE]

Although many different hypotheses sets exist, $\mathcal{H}$ can be chosen as described in [13] as a linear classifier defined as follows,

[TABLE]

The solution to this problem boils down to solving the following convex optimization problem:

[TABLE]

where for any integer $m$, $[1:m]$ denotes $\{1,\ldots,m\}$. This notation is used throughout this paper. The above problem can be solved by introducing Lagrange variables $\alpha_{i}\geq 0,i\in[1:m]$ for each constraint. Thus, the dual form of the constrained optimization problem is derived as following.

[TABLE]

Solving the dual problem on $\alpha_{i},i\in[1:m]$, we have:

[TABLE]

As is clear from (5)-(7), in order to solve the main problem for $\mathbf{w},b$, we only need the inner products of samples and their labels to solve the dual problem for $\alpha_{i}$ and a linear combination of data samples to get $\mathbf{w}$ 111To having a linear combination of the samples privately, we can use a scheme called private function retrieval. So, when the length of vectors $\mathbf{x}_{i}$, $L$, is large, retrieving inner products instead of raw samples is more efficient in a distributed learning setting.

2. Regression: The regression algorithm predicts the real-valued label of a point by using a data set. Regression is a very common task in machine learning for approximately and closely deriving the relationship between variables. The regression is similar to continuous-label version of the classification, as opposed to the classification’s discrete labels. Many use cases can be considered for the regression algorithm, such as optimizing the price of products by learning the relation of price and the sale volume in different markets and analyzing the product sale drivers such as distribution methods in markets. Here, we first describe a simple regression problem from [13, Page 245] and show in order to solve this problem we only need the inner products as opposed to retrieve all data files.

Similar to SVM, consider an input alphabet $\mathcal{X}$ consisting of vectors of length $L$ and a distribution $\mathcal{D}$ on $\mathcal{X}\times\mathcal{Y}$. The learner has $m$ training samples from $\mathcal{X}\times\mathcal{Y}$, denoted by $(\mathbf{x}_{1},y_{1}),(\mathbf{x}_{2},y_{2}),\ldots,(\mathbf{x}_{m},y_{m})$, drawn from $\mathcal{D}$. The difference is that the target output alphabet $\mathcal{Y}$ can be a continuous space. Since the labels are real numbers, the learner is not able to predict them precisely. So, a loss function is considered to show the distance between the label and the predicted value.

Now, we discuss a simple linear regression problem. Similar to SVM, the hypothesis set is as follows.

[TABLE]

The loss here is empirical mean squared error. So, the optimization problem is as follows,

[TABLE]

which can be written in a simpler form as:

[TABLE]

where $\mathbf{X}=\begin{bmatrix}\mathbf{x_{1}}&...&\mathbf{x_{m}}\\ 1&...&1\end{bmatrix}$, $\tilde{\mathbf{w}}=\begin{bmatrix}w_{1}\\ \vdots\\ w_{L}\\ 1\end{bmatrix}$ and $\mathbf{y}=\begin{bmatrix}y_{1}\\ \vdots\\ y_{m}\end{bmatrix}$. It is clear that the objective functions is convex and reaches its optimum value in $\nabla F(\tilde{\mathbf{w}})=0$. So, we have:

[TABLE]

Now, if $\mathbf{X}\mathbf{X}^{\top}$ is invertible, we can calculate $\tilde{\mathbf{w}}$. Otherwise, we replace the inverse with pseudo-inverse.

[TABLE]

It can be easily shown that the above result can be rewritten as below.

[TABLE]

As seen, the solution only needs inner products ($\mathbf{X}^{\top}\mathbf{X}$) and a linear combination of data files ($\tilde{\mathbf{w}}=\mathbf{X}\mathbf{a},$ $\mathbf{a}=(\mathbf{X}^{\top}\mathbf{X})^{-1}\mathbf{y}$) and not all data files. If the length of data vectors, $L$, is large, downloading all data files needs much more resource.

3. Principal component analysis (PCA): The purpose of this algorithm is to reduce the dimensionality of data with large vector length, so that its most important features can be better analyzed. The reason is that sometimes the generalization ability of method decreases with the increase in dimension of data. The following example is from [14, Page 324].

Consider the $m$ vectors of length $L$, $\mathbf{x}_{1},...,\mathbf{x}_{m}$, as data files. The goal here is to reduce the dimensionality of these vectors using linear transformation. To do this, we define a matrix $\mathbf{W}\in\mathbb{R}^{d\times L}$ where $d<L$. We also have a mapping $\mathbf{x}\mapsto\mathbf{Wx}$, whose output is the lower dimensionality representation of data. Then a second matrix $\mathbf{U}\in\mathbb{R}^{L\times d}$ is defined to recover $\mathbf{x}$. This means that if $\mathbf{y}=\mathbf{Wx}$ is the reduced representation, then the $\tilde{\mathbf{x}}=\mathbf{Uy}$ is the recovered data. Minimizing the magnitude of empirical distance between the original data and the recovered data is the goal of PCA.

[TABLE]

It is shown in [14] that $\mathbf{U}^{\top}=\mathbf{W}$ and this problem can be rewritten as follows.

[TABLE]

where $\mathbf{I}$ is the identity matrix. According to Theorem 23.2 in [14, Page 325] the solution for above problem is to calculate $\mathbf{u}_{1},...,\mathbf{u}_{d}$ which are eigenvectors of matrix $\mathbf{A}=\sum_{i=1}^{m}\mathbf{X}\mathbf{X}^{\top}$ ($\mathbf{X}=[\mathbf{x_{1}}...\mathbf{x_{m}}]$) corresponding to $d$ largest eigenvalues of the matrix. The solution is $\mathbf{U}=[\mathbf{u}_{1}...\mathbf{u}_{d}]$.

If the dimension of the original vectors is too large ($L\gg m$), then we can rewrite the answer. We define $\mathbf{B}=\mathbf{X}^{\top}\mathbf{X}$. Let $\mathbf{u}$ be an eigenvector of matrix $\mathbf{B}$ (so $\mathbf{B}\mathbf{u}=\lambda\mathbf{u}$). This means that we have $\mathbf{X}^{\top}\mathbf{X}\mathbf{u}=\lambda\mathbf{u}$ and thus,

[TABLE]

Therefore, if $\mathbf{u}$ is an eigenvector of $\mathbf{B}$, corresponding to eigenvalue $\lambda$, then $\mathbf{Xu}$ is an eigenvector of matrix $\mathbf{A}$, corresponding to the same eigenvalue. So in PCA, when vector length $L$ is large, it is simpler to calculate the matrix $\mathbf{X}^{\top}\mathbf{X}$ that is matrix of inner products of original data. Then, the eigenvectors of this matrix corresponding to its $d$ largest eigenvalues are enough.

These three algorithms make clear that methods using inner products of data files are important and common tasks of machine learning. Thus, retrieving the inner products privately from the servers is an important step in machine learning privacy.

III Problem Statement

Consider a set of $K$ data files, $W_{1},\ldots,W_{K}$, for some integer $K$, where files are selected independently and uniformly at random from a finite field $\mathbb{F}(q^{L})$, for some integer $L$. Thus,

[TABLE]

Files can be represented in the vector form as

[TABLE]

We assume that files are replicated in $N$ non-colluding servers, for some integer $N$. We define $\mathcal{X}^{(L)}$, as the set of the inner product of all pairs of data files,

[TABLE]

Also, we define $\mathcal{T}$ as index of inner products as follows,

[TABLE]

Note that each member of $\mathcal{T}$ corresponds to an inner product in set $\mathcal{X}^{(L)}$, i.e., $\{i,j\}\in\mathcal{T}\iff\langle W_{i},W_{j}\rangle\in\mathcal{X}^{(L)}$.

A user wishes to retrieve a subset of size $P\in\mathbb{N}$ of inner products. More precisely, the user chooses a set $\mathcal{P}$, where $\mathcal{P}\subseteq\mathcal{T}$, and $|\mathcal{P}|=P$, and entreats to know $\mathcal{X}^{(L)}_{\mathcal{P}}$, defined as

[TABLE]

The cardinality $P$ of $\mathcal{P}$ is known to all servers. The user wishes to retrieve $\mathcal{X}^{(L)}_{\mathcal{P}}$ while ensuring privacy of $\mathcal{P}$ from each server.

In order to retrieve these inner products user creates queries $Q_{1}^{[\mathcal{P}]},...,Q_{N}^{[\mathcal{P}]}$ and sends $Q_{n}^{[\mathcal{P}]}$ to server $n$, through an error-free secure link. In response, server $n$, responds with $A_{n}^{[\mathcal{P}]}$. Since user has no knowledge of files,

[TABLE]

The answer of server $n$, $n\in[1:N]$, is a function of query sent to that server and the set of data files available there, thus

[TABLE]

Also, $A_{n_{1}:n_{2}}$ denotes set $\{A_{n_{1}},A_{n_{1}+1},...,A_{n_{2}}\}$. The queries and answers must satisfy two conditions:

(i) Correctness Condition: This condition states that by having all queries and answers from servers, the user can calculate inner products indexed by the set $\mathcal{P}$. Equivalently,

[TABLE]

(ii) Privacy Condition: In order to satisfy privacy, regardless of what set $\mathcal{P}$ is chosen, query and answer for each server must be identically distributed, i.e., $\forall\mathcal{P}_{1},\mathcal{P}_{2}\subseteq\mathcal{T}$ , $|\mathcal{P}_{1}|=|\mathcal{P}_{2}|=P$, we must have,

[TABLE]

For an achievable scheme, satisfying (25) and (26), we define the retrieval rate $R(P,L)$, as the ratio between information of the inner products in $\mathcal{X}^{(L)}_{\mathcal{P}}$ and total downloading cost to retrieve the inner products $\mathcal{X}^{(L)}_{\mathcal{P}}$, minimized over all possible requests $\mathcal{P}\subseteq\mathcal{T}$, $|\mathcal{P}|=P$, i.e.,

[TABLE]

The capacity is the supremum of all achievable $R(P,L)$.

IV Preliminary

In order to proceed we need to review the results of MPIR problem in [7]. Consider a system, including $K$ data files, replicated in $N$ noncoluding servers. Each data file is chosen independently and uniformly at random from the finite field $\mathbb{F}(q^{L})$. A user wishes to retrieve a subset indexed by $\mathcal{P}\subseteq[1:K]$ of data files, ensuring the privacy of $\mathcal{P}$. Assume $|\mathcal{P}|=P$, where $P$ is known publicly. Rate is defined as information of subset of data files indexed by $\mathcal{P}$ over download cost, and the capacity $C_{\mathsf{MPIR}}$ is defined as the supremum over all rates in privacy preserving schemes. Then we have [7],

[TABLE]

where for $\frac{K}{P}\leq 2$, we have

[TABLE]

and for $\frac{K}{P}\geq 2$, we have

[TABLE]

and $\dfrac{1}{\underline{R}_{\mathsf{MPIR}}(K,P,N)}$ is equal to

[TABLE]

where $r_{i}$ is defined as $r_{i}=\dfrac{e^{\hat{j}2\pi(i-1)/P}}{N^{1/P}-e^{\hat{j}2\pi(i-1)/P}},i\in[1:P],$ and $\hat{j}=\sqrt{-1}$. In addition, $\beta_{i},i\in[1:P]$, is the solution of the set of linear equations $\sum_{i=1}^{P}\beta_{i}r_{i}^{-P}=(N-1)^{K-P}$ and $\sum_{i=1}^{P}\beta_{i}r_{i}^{-k}=0$, $k\in[1:P-1]$.

V Main Results

The main result is stated in the following theorem.

Theorem 1**.**

For a system with $K$ files in $\mathbb{F}(q^{L})$ and $N$ servers, where the user is interested in a subset of size $P$ of inner products, we have

[TABLE]

where $\lambda_{2}$ is a constant independent of $L$ and $|\lambda_{2}|<1$.

Corollary 1**.**

If $\dfrac{K(K+1)}{2P}\leq 2$, then we have

[TABLE]

Corollary 2**.**

If $\dfrac{K(K+1)}{2P}\in\mathbb{N}$, then we have

[TABLE]

The proof can be found in the next section. Assuming $q$ is large enough, for achievability, we use the scheme of MPIR. For converse, we prove that as $L$ goes to infinity, entries of $\mathcal{X}^{(L)}$ converges to a set of independent random variables with uniform distribution, with the rate of convergence dominated by a constant $\lambda_{2}$, $|\lambda_{2}|\leq 1$. For large $L$, in some cases, the achievable rate and converse match. In other cases, these two are very close.

VI Proof

We sort the elements of set $\mathcal{X}^{(L)}$ in a vector $X^{(L)}\in\mathbb{F}^{K(K+1)/2}(q)$, such that $\langle W_{i},W_{j}\rangle$ in $X^{(L)}$ comes before $\langle W_{k},W_{l}\rangle$ if $i<k$ or $i=k$ and $j<l$. Likewise, we sort the elements of $\mathcal{X}^{(L)}_{\mathcal{P}}$ in a vector $X^{(L)}_{\mathcal{P}}$.

In this section, we provide the proof for Theorem 1.

First we show that as $L\to\infty$, the distribution of $X^{(L)}$ converges to a uniform distribution over $\mathbb{F}^{K(K+1)/2}(q)$:

[TABLE]

Indeed, we increase $L$ by one, and show that the distribution of $X^{(L)}$ over $\mathbb{F}^{K(K+1)/2}(q)$ becomes closer to a uniform distribution. In addition, we derive the rate of convergence.

Let us denote the $q^{K(K+1)/2}$ members of set $\mathbb{F}^{K(K+1)/2}(q)$ by $\mathbf{y}_{1}\ ...\ \mathbf{y}_{q^{K(K+1)/2}}$, i.e.,

[TABLE]

We denote the probability mass function of $X^{(L)}$ over $\mathbb{F}^{K(K+1)/2}(q)$ by $\mathbf{p}^{(L)}\in[0,1]^{q^{K(K+1)/2}}$, i.e.

[TABLE]

where

[TABLE]

Apparently,

[TABLE]

Our goal is to investigate how $\mathbf{p}^{(L)}$ changes, as we increase $L$ to $L+1$. Let

[TABLE]

Without loss of generality, we assume that

[TABLE]

where $w_{i(L+1)}$ is selected uniformly at random from $\mathbb{F}(q)$. We note that by this construction $X^{(L)}$ and $X^{(L+1)}$ become correlated. However, the distribution of $X^{(L+1)}$ is still the same as it was discussed in the problem formulation but this correlation allows us to derive the converging distribution.

Lemma 1**.**

The sequence ${\{X^{(L)}\}}_{L=1}^{\infty}$ forms a Markov chain with a time-homogeneous transition probability $\mathbf{M}\in\mathbb{R}^{q^{K(K+1)/2}\times q^{K(K+1)/2}}$, i.e.

[TABLE]

where

[TABLE]

Proof.

Defining the data files as above, then we have,

[TABLE]

Thus for the vector of inner products $X^{(L)}$, we also can write,

[TABLE]

where

[TABLE]

Because of the way we constructed $W_{i}^{(L+1)}$ from $W_{i}^{(L)}$, for $i=[1:K]$, it is apparent that $\Delta^{(L+1)}$ is independent of data files $W_{i}^{(L)}$, $i\in[1:K]$, and irrespective of $L$. We have

[TABLE]

Thus from (35), we can rewrite the above equation as

[TABLE]

where $\mathbf{M}\in\mathbb{R}^{q^{K(K+1)/2}\times q^{K(K+1)/2}}$ is a constant matrix, with entry $(i,j)$ be equal to

[TABLE]

We note that $\mathbf{M}$ is constant and independent of $L$. ∎

To show that the limit in (32) exists, in the following lemma, we guarantee that the Markov chain has steady distribution.

Lemma 2**.**

Markov chain formed by the sequence ${\{X^{(L)}\}}_{L=1}^{\infty}$ is irreducible.

Proof.

In order to prove lemma we show that there exists some $\Gamma\in\mathbb{N}$, such that $[\mathbf{M}^{\Gamma}]_{i,j}>0$, $\forall i,j\in[1:q^{K(K+1)/2}]$. This means it is possible to get to any state from any state in this chain or equivalently this chain is irreducible. We note that for any integer $\Gamma$

[TABLE]

where

[TABLE]

One can see that

[TABLE]

$[\mathbf{M}^{\Gamma}]_{i,j}$ denotes entry $(i,j)$ of matrix $\mathbf{M}^{\Gamma}$.

This lemma is equivalent to claim that there exists some $\Gamma\in\mathbb{N}$ such that every realization of $\Delta^{(L,L+\Gamma)}$ in $\mathbb{F}^{K(K+1)/2}(q)$ is possible with some positive probability. Notice that the following relationship holds,

[TABLE]

It is obvious that $\Delta^{(L+\gamma-1,L+\gamma)}$, $\gamma=1,\ldots,\Gamma$, are mutually independent. The reason is that $\Delta^{(L+\gamma-1,L+\gamma)}$ is only dependent of $w_{k(L+\gamma)},k\in[1:K]$.

We first show that for $\Gamma=5$ every vector in $\mathbb{F}^{K(K+1)/2}(q)$ with only one non-zero element is a probable (has a positive probability) realization of $\Delta^{(L,L+5)}$. In other words, we show that, for any $\mathbf{y}\in\mathbb{F}^{K(K+1)/2}(q)$, where $\mathbf{y}(e)\neq 0$ and $\mathbf{y}(i)=0$, $\forall i\in[1:q^{K(K+1)/2}]\backslash\{e\}$, for some $e\in[1:q^{K(K+1)/2}]$, then $\Pr\{\Delta^{(L,L+5)}=\mathbf{y}\}>0$. Let us assume

[TABLE]

We know that by definition $\Delta^{(L,L+5)}(e)=\sum_{\gamma=1}^{5}w_{i_{e}(L+\gamma)}.w_{j_{e}(L+\gamma)}$, for some $i_{e},j_{e}\in[1:K]$. Here, we consider two cases for values of $i_{e},j_{e}$.

• Case (I)

: In this case $i_{e}=j_{e}$. In other words, $\Delta^{(L,L+5)}(e)=\sum_{\gamma=1}^{5}w_{i_{e}(L+\gamma)}.w_{i_{e}(L+\gamma)}$, for some $i_{e}\in[1:K]$

From [15, Page 66 ], we have

[TABLE]

Therefore one possible case that can create such $\mathbf{y}$ is as follows:

[TABLE]

Clearly this case has positive probability and therefore $\Pr\{\Delta^{(L,L+\Gamma)}=\mathbf{y}\}>0$.

• Case (II):

In the case $i_{e}\neq j_{e}$. In other words, $\Delta^{(L,L+5)}(e)=\sum_{\gamma=1}^{5}w_{i_{e}(L+\gamma)}.w_{j_{e}(L+\gamma)}$, for some $i_{e},j_{e}\in[1:K]$, $i_{e}\neq j_{e}$.

We have (see [15, Page 66])

[TABLE]

Therefore one possible case that can create such $\mathbf{y}$ is as follows:

[TABLE]

In particular, one can verify that

[TABLE]

In addition

[TABLE]

[TABLE]

Other entries of $\Delta^{(L,L+5)}$ are zero trivially.

Since the probability of (55) is not zero, therefore in this case also $\Pr\{\Delta^{(L,L+\Gamma)}=\mathbf{y}\}>0$.

From these two cases above, we can say every vector with one non-zero element is a probable (with positive probability) realization of $\Delta^{(L,L+5)}$. We now show that every vector in $\mathbb{F}^{K(K+1)/2}(q)$ is a possible realization with positive probability for $\Delta^{(L,L+\Gamma)}$ when $\Gamma\geq 5K(K+1)/2$. First we write $\Delta^{(L,L+5K(K+1)/2)}$ as,

[TABLE]

Let $\mathbf{y}\in\mathbb{F}^{K(K+1)/2}(q)$ be an arbitrary vector. To show that $\mathbf{y}$ is a possible realization of $\Delta^{(L,L+\Gamma)}$ with non-zero probability, we first define $\mathbf{y}^{(i)},i\in[1:K(K+1)/2]$, as follows,

[TABLE]

This means $\mathbf{y}^{(i)}$ is zero in every index except in index $i$ where its value is $\mathbf{y}(i)$. We can see that,

[TABLE]

By construction, $\mathbf{y}^{(i)}$ is a vector that has at most one non-zero element, thus it is a probable realization for $\Delta^{(L+5(i-1),L+5(i))}$. Now if $\Delta^{(L+5(i-1),L+5(i))}=\mathbf{y}^{(i)}$, ,$\forall i\in[1:K(K+1)/2]$ which is possible with positive probability then because of (62) and (59), we know $\Delta^{(L,L+5K(K+1)/2)}=\mathbf{y}$, therefore $\Pr\{\Delta^{(L,L+5K(K+1)/2)}=\mathbf{y}\}>0$. Also because of (49) every element in the matrix $\mathbf{M}^{5K(K+1)/2}$ is positive. ∎

Corollary 3**.**

Markov sequence ${\{X^{(L)}\}}_{L=1}^{\infty}$ has a steady state.

Proof.

Markov sequence ${\{X^{(L)}\}}_{L=1}^{\infty}$ has a unique steady state if there exist an integer $\Gamma$ that $\mathbf{M}^{\Gamma}$ has an all positive row  [16, Page 176], as it is proved in Lemma 2. ∎

Lemma 3**.**

As $L\rightarrow\infty$, Markov chain ${\{X^{(L)}\}}_{L=1}^{\infty}$ converges to a random vector with uniform distribution over $\mathbb{F}^{K(K+1)/2}(q)$.

Proof.

It is known that if a Markov chain has steady state, its stationary distribution is equal to its steady state probabilities  [16, Page 174]. We use this fact to find that steady state. As obtained, we know $[\mathbf{M}]_{i,j}=\Pr\{\Delta^{(L,L+1)}=\mathbf{y}_{j}-\mathbf{y}_{i}\}$. It is easy to see that for any $i$, the set $\{\mathbf{y}_{i}-\mathbf{y}_{j},j\in[1:q^{K(K+1)/2}]\}$ is equal to $\mathbb{F}^{K(K+1)/2}(q)$. Thus,

[TABLE]

Let $\bm{\pi}=(1/q^{K(K+1)/2},...,1/q^{K(K+1)/2})^{\top}$. It is easy to see that due to (63), $\mathbf{M}\bm{\pi}=\bm{\pi}$. Thus uniform distribution is stationary state probability of this Markov chain. ∎

Lemma 4**.**

Let $\mathbf{p}^{(L)}\in[0,1]^{q^{K(K+1)/2}}$ denote the PMF of $X^{(L)}$ over $\mathbb{F}^{K(K+1)/2}(q)$. Then, $\|\mathbf{p}^{(L)}-\bm{\pi}\|_{\infty}=O(\lambda_{2}^{L-1})$, where $\lambda_{2}$ is the second largest eigenvalue (absolute value of eigenvalue) of $\mathbf{M}$ and $|\lambda_{2}|<1$.

We show that,

[TABLE]

where $\bm{\pi}=(\dfrac{1}{q^{K(K+1)/2}},...,\dfrac{1}{q^{K(K+1)/2}})^{\top}$ and $\mathbf{1}=(1,...,1)^{\top}\in\mathbb{R}^{K(K+1)/2}$.

From Lemma 1, we know that $\{X^{(L)}\}_{L=1}^{\infty}$ forms a Markov chain with transition matrix $\mathbf{M}$. Thus, the PMF of $X^{(L)}$, denoted by $\mathbf{p}^{(L)}$, is equal to

[TABLE]

where $\mathbf{p}^{(1)}$ is the PMF of $X^{(1)}$. Also from Lemma 3, we know that this Markov chain has a steady state. Thus for the eigenvalue of transition matrix $\mathbf{M}$, we have,

[TABLE]

As stated, matrix $\mathbf{M}$ and thus its eigenvalues are independent of $L$.

Let $\bm{\pi},\mathbf{v}_{2},\ldots,\mathbf{v}_{q^{K(K+1)/2}}\in\mathbb{R}^{q^{K(K+1)/2}}$ denote the right eigenvectors corresponding to the eigenvalues $\lambda_{1},\lambda_{2},\ldots,\lambda_{q^{K(K+1)/2}}$, respectively. We note that $\bm{\pi},\mathbf{v}_{2},\ldots,\mathbf{v}_{q^{K(K+1)/2}}$ forms an orthogonal bases for $\mathbb{R}^{q^{K(K+1)/2}}$. Thus, we can expand $\mathbf{p}^{(1)}$ as

[TABLE]

for some $\alpha_{i}$, $i\in[1:{q^{K(K+1)/2}}]$.

Therefore we can write,

[TABLE]

From the fact that $\lim_{L\rightarrow\infty}\mathbf{p}^{(L)}=\bm{\pi}$ and also (66) we know when $L\to\infty$ every term in (68) diminishes except $\lambda_{1}^{L-1}\alpha_{1}\bm{\pi}$ (where $\lambda_{1}=1$) which should be equal to $\bm{\pi}$. Thus we can rewrite (68) as,

[TABLE]

Thus for every element of $\mathbf{p}^{(L)}$ from (69), we have,

[TABLE]

Lemma 5**.**

Entropy of set $\mathcal{X}^{(L)}_{\mathcal{P}}$ of inner products when $|\mathcal{P}|=P$ satisfies,

[TABLE]

Proof.

This lemma gives a lower bound on entropy of inner products indexed by a subset $\mathcal{P}$. We first calculate probability distribution of $X^{(L)}_{\mathcal{P}}$ over $\mathbb{F}^{P}(q)$. For the set $\mathcal{P}$, we define the vector of indices $\bm{\tau}_{\mathcal{P}}$, such that $X^{(L)}_{\mathcal{P}}(i)=X^{(L)}(\tau_{\mathcal{P}}(i))$, for $i=1,\ldots,P$. In addition, for a $\mathbf{z}\in\mathbb{F}^{P}(q)$, we define

[TABLE]

It is easy to see that $|\mathcal{S}_{\mathbf{z}}|=q^{K(K+1)/2-P}$. Now we can calculate the probability distribution of $X^{(L)}_{\mathcal{P}}$ over $\mathbb{F}^{P}(q)$.

[TABLE]

where (75) is the result of Lemma 4.

For entropy of inner products in set $\mathcal{X}^{(L)}_{\mathcal{P}}$, we can write,

[TABLE]

∎

Now that we have the result of lemma 5 and by employing MPIR achievable scheme on inner products we are able to achieve result of theorem 1.

VI-A Proof of Theorem 1

This proof is similar to the proof in [7] changed to match the current problem setting. To show the limits of capacity, this proof needs to be split in two parts, achievability and converse.

Achievability: We use the same achievability scheme that is used in MPIR problem. In this case as the problem setting states, user wants to privately retrieve all inner products included in $\mathcal{X}^{(L)}_{\mathcal{P}}$ without revealing the identity of $\mathcal{P}$. Now we treat every inner product like an entry data file in MPIR and run the proposed scheme on them. Notice that by running MPIR scheme, inner products indexed by the mentioned subset are privately retrieved and to servers all subsets of size $P$ are equiprobable so privacy constraint is met and we can say,

[TABLE]

Here the number of data files in MPIR is the number of inner products, i.e. $K(K+1)/2$.

Converse: In order to prove the converse for Theorem 1 (i.e., to derive an upper bound on the capacity), we use Lemma 5 from which we have a lower bound on the entropy of an arbitrary subset of inner products $\mathcal{P}$ as,

[TABLE]

To continue with the proof, we consider the problem in two cases, $\dfrac{K(K+1)}{2P}\leq 2$ and $\dfrac{K(K+1)}{2P}>2$.

Case 1: $\dfrac{K(K+1)}{2P}\leq 2$.

Remember that the user sends queries $Q_{n}^{[\mathcal{P}]},n\in[1:N]$, to servers and receives answers $A_{n}^{[\mathcal{P}]},n\in[1:N]$. We define the set of all possible queries as

[TABLE]

and also the set of answers from servers $n_{1}$ to $n_{2}$ as,

[TABLE]

Note that the number of all possible $\mathcal{P}$ is $\beta=\binom{K(K+1)/2}{P}$.

We can assume symmetry in the scheme across data files and servers queries and answers. Even if the scheme is asymmetric we can replicate scheme for every permutation of data bases and servers and create a symmetric scheme.

Since the queries and answers are independent of the desired set because of (26), we fix the answers of server 1 to be (same as the MPIR),

[TABLE]

The proof of the following lemma is similar to the proof of [7, Lemma 1].

Lemma 6**.**

For any $\mathcal{S}\subseteq\mathcal{T}$ and $\mathcal{X}^{(L)}_{\mathcal{S}}=\{\langle W_{i},W_{j}\rangle|\{i,j\}\in\mathcal{S}\}$, we have,

[TABLE]

(87) is the result of symmetry assumption and (26).

First, we derive a lower bound on the entropy of inner products appeared in $A_{1}^{[\mathcal{P}]}$, that are not in $\mathcal{X}^{(L)}_{\mathcal{P}}$.

Lemma 7**.**

For the problem stated in Section III, with $P\geq\dfrac{K(K+1)}{2}$, the following lower bound on the conditional entropy of $A_{1}^{[\mathcal{P}]}$ holds.

[TABLE]

Proof.

Here we define $\bar{\mathcal{P}}\subseteq\mathcal{T}$ to be the set with size $|\bar{\mathcal{P}}|=P$ such that $\mathcal{P}\cup\bar{\mathcal{P}}=\mathcal{T}$.

Such a set $\bar{\mathcal{P}}$ exists, since $\mathcal{P}$ has more than the half of all inner products (due to $P\geq K(K+1)/4$). Now, we use (82) for $\mathcal{X}^{(L)}$ to write the following.

[TABLE]

where (91) follows from (25) noting that $\mathcal{X}^{(L)}\setminus\mathcal{X}^{(L)}_{\mathcal{P}}\subseteq\mathcal{X}^{(L)}_{\bar{\mathcal{P}}}$; (93) is true because of (24); and (95) is the result of Lemma 6. ∎

Now, we proceed to the converse following the approach of [7] as,

[TABLE]

where (97) follows (23); (98) follows from (25) and the fact that $\mathcal{P}_{1},\ldots,\mathcal{P}_{\beta}$ are distinct sets that cover all possible inner products and thus every inner product can be decoded from $A_{1:N}^{[\mathcal{P}_{1}]},...,A_{1:N}^{[\mathcal{P}_{\beta}]}$; (100) holds thanks to (24); (101) is true because of (85); and (103) is due to (25).

We can see that,

[TABLE]

where (106) is true because of (24) and the last inequality follows from (82). Now, combining the result of Lemma 7 with (105) and (108) results in:

[TABLE]

which can be written as,

[TABLE]

and further is simplified as,

[TABLE]

which completes the converse for Case 1 as:

[TABLE]

Case 2: $\dfrac{K(K+1)}{2P}>2$.

Here, similar to the proof of MPIR, we create an inductive relation and use the result of Case 1 as the base induction step. In order to continue with the induction proof we need to introduce a slightly different problem formulation (we call as modified problem). Assume that the user wishes to retrieve inner products indexed by the set $\mathcal{P}_{1}$ while ensuring privacy over $\forall\mathcal{P}\subseteq\mathcal{T}_{1}\subseteq\mathcal{T}$ where $|\mathcal{T}_{1}|=T_{1}$ and $\mathcal{T}_{1}$ is fixed and known to all. We denote the query set in this problem by $\hat{\mathcal{Q}}$ and the answer from $n$-th server by $\hat{A}^{[\mathcal{P}]}_{n}$. Number of all possible sets $\mathcal{P}$ is $\hat{\beta}=\binom{T_{1}}{P}$.

Note that for $\mathcal{T}_{1}=\mathcal{T}$, the modified problem reduces to the original problem (of Section III). The reason for introducing the modified problem is that $|\mathcal{T}|$ in the original problem is equal to $|\mathcal{T}|=K(K+1)/2$ (that is $|\mathcal{T}|$ cannot take any arbitrary integer). However, for the inductive step, we need the number of inner products to take any integer.

It is easy to see that the result of case 1 is still true for the modified problem. Hence, when $T_{1}/P\leq 2$ we have,

[TABLE]

We use this result as our induction base step. Now we proceed to prove that for case $T_{1}/P>2$ we have,

[TABLE]

We achieve (115) by induction with (114) as induction base. First, we provide a lemma to derive an upper bound on the remaining information in answer $\hat{A}_{2:N}^{[\mathcal{P}_{2}]}$ conditioned on inner products indexed by the set $\mathcal{P}_{1}\subseteq\mathcal{T}_{1}$ when $\mathcal{P}_{1}$ and $\mathcal{P}_{2}$ have the same size but have no similar inner product in their corresponding sets.

Lemma 8**.**

For two sets $\mathcal{P}_{1}\subseteq\mathcal{T}_{1}$ and $\mathcal{P}_{2}\subseteq\mathcal{T}_{1}$, if $|\mathcal{P}_{1}|=|\mathcal{P}_{2}|=P$ and $\mathcal{P}_{1}\cap\mathcal{P}_{2}=\phi$, then the following inequality holds.

[TABLE]

Proof.

[TABLE]

where (120) holds because of independence of queries and data files; and (121) follows from (82), the symmetry across the servers and the fact that $\mathcal{X}^{(L)}_{\mathcal{P}_{1}}$ can be calculated from $\hat{A}_{1:n-1}^{[\mathcal{P}_{1}]},\hat{A}_{n}^{[\mathcal{P}_{2}]},\hat{A}_{n+1:N}^{[\mathcal{P}_{1}]}$ and queries which is the result of (25). ∎

Now we construct the inductive step, similar to MPIR but tailored to our setting. Assume that two subsets $\mathcal{P}_{1}$ and $\mathcal{P}_{2}$ are chosen from all subsets $\mathcal{P}_{i}\subseteq\mathcal{T}_{1},i\in[1:\hat{\beta}],|\mathcal{P}|=P$ such that $\mathcal{P}_{1}\cap\mathcal{P}_{2}=\phi$. We see,

[TABLE]

where (123) follows from the fact that knowing $A_{1},A_{2:N}^{[\mathcal{P}_{1}]},...,A_{2:N}^{[\mathcal{P}_{\hat{\beta}}]}$ one can obtain all inner products indexed by $\mathcal{T}_{1}$ (noting that it is always possible to fix the answers of one of the servers (to be independent of the desired set) as done in (85)); (125) is true due to the symmetry across the servers and the fact that $\mathcal{X}^{(L)}_{\mathcal{P}_{1}}$ and $\mathcal{X}^{(L)}_{\mathcal{P}_{2}}$ have no more information if we have $A_{1},A_{2:N}^{[\mathcal{P}_{1}]},A_{2:N}^{[\mathcal{P}_{2}]}$ and queries which is the result of (25); (128) is true because of (24) similar to (106); and (130) follows from (82).

We rewrite (131) as,

[TABLE]

which can also be written as,

[TABLE]

Now, as mentioned in [7], $H(\hat{A}_{1}|\mathcal{X}^{(L)}_{\mathcal{P}_{1}},\mathcal{X}^{(L)}_{\mathcal{P}_{2}},\hat{\mathcal{Q}})$ (in the modified problem) is quite similar to the $H(\hat{A}_{1}|\hat{\mathcal{Q}})$ in an equivalent problem where the user wants to retrieve a subset of inner products indexed by $\mathcal{P}\subseteq\mathcal{T}_{2}$ and $\mathcal{T}_{2}=\mathcal{T}_{1}\setminus(\mathcal{P}_{1}\cup\mathcal{P}_{2})$ and thus $|\mathcal{T}_{2}|=T_{1}-2P$. The difference between the modified problem and its equivalent problem is that in the modified problem we have conditions on $\mathcal{X}^{(L)}_{\mathcal{P}_{1}},\mathcal{X}^{(L)}_{\mathcal{P}_{2}}$, while in the equivalent problem the conditions don’t exist. These two problem would be completely equivalent if the inner products were mutually independent. However, by adding the conditions, one can obtain all the equations up to (133) for the equivalent problem with a difference of $O(\lambda_{2}^{L-1})$. Therefore, we can say $H(\hat{A}_{1}|\mathcal{X}^{(L)}_{\mathcal{P}_{1}},\mathcal{X}^{(L)}_{\mathcal{P}_{2}},\hat{\mathcal{Q}})$ in the modified problem with total number of inner products $|\mathcal{T}_{1}|=T_{1}$ is equal to $H(\hat{A}_{1}|\hat{\mathcal{Q}})$ in an equivalent problem with total number of inner products $|\mathcal{T}_{1}|=T_{1}-2P$. We use this result in our induction step.

Now, we start with the following induction hypothesis when the number of all inner products is $|\mathcal{T}_{1}|=T_{1}-2P+1$:

[TABLE]

To complete the proof by induction we must show that (134) holds for $|\mathcal{T}_{1}|=T_{1}+1$.

Using the equivalency of the modified and the equivalent problem, (134) (which was written for the equivalent problem with $|\mathcal{T}_{1}|=T_{1}-2P+1$) is true for the modified problem if we substitute $H(\hat{A}_{1}|\mathcal{X}^{(L)}_{\mathcal{P}_{1}},\mathcal{X}^{(L)}_{\mathcal{P}_{2}},\hat{\mathcal{Q}})$ by $H(\hat{A}_{1}|\hat{\mathcal{Q}})$ (as well as $T_{1}-2P+1$ by $T_{1}+1$) to obtain:

[TABLE]

Combining (133) and (137) results in,

[TABLE]

From induction hypothesis (134), we have proven inductive step in (139) and thus (115) is true. As mentioned before, when $\mathcal{T}_{1}=\mathcal{T}$ or equivalently $T_{1}=K(K+1)/2$, the modified problem reduces to the original problem. Therefore (115) holds for the original problem. Therefore, we have,

[TABLE]

From the results of two cases (113) and (143) we can write,

[TABLE]

Comparing (28) with the results of converse, (144), and achievability, (81), the proof of Theorem 1 is complete.