On the Strong Restricted Isometry Property of Bernoulli Random Matrices

TL;DR

This paper demonstrates that Bernoulli random matrices satisfy the strong restricted isometry property (SRIP) with high probability, ensuring robustness in compressed sensing even after row erasures, and extends the Johnson-Lindenstrauss lemma.

Contribution

It proves Bernoulli matrices satisfy SRIP with high probability, enhancing understanding of their robustness in compressed sensing applications.

Findings

Bernoulli matrices satisfy SRIP with overwhelming probability.

The analysis extends the Johnson-Lindenstrauss lemma to Bernoulli matrices.

Results confirm robustness of Bernoulli matrices under row erasures.

Abstract

The study of the restricted isometry property (RIP) of corrupted random matrices is particularly important in the field of compressed sensing (CS) with corruptions. If a matrix still satisfies the RIP after that a certain portion of rows are erased, then we say that this matrix has the strong restricted isometry property (SRIP). In the field of compressed sensing, random matrices which satisfy certain moment conditions are of particular interest. Among these matrices, those with entries generated from i.i.d. Gaussian or symmetric Bernoulli random variables are often typically considered. Recent studies have shown that matrices with entries generated from i.i.d. Gaussian random variables satisfy the SRIP under arbitrary erasure of rows with high probability. In this paper, we study the erasure robustness property of Bernoulli random matrices. Our main result shows that with overwhelming probability, the SRIP holds for Bernoulli random matrices. Moreover, our analysis leads to a robust version of the famous Johnson-Lindenstrauss lemma for Bernoulli random matrices.

Equations

$$(1-\delta )\Vert x{\Vert }_2^2\le \Vert Ax{\Vert }_2^2\le (1+\delta )\Vert x{\Vert }_2^2$$

$${\delta }_{s,A}:=\inf \{\delta \in [0,1):(1-\delta )\Vert x{\Vert }_2^2\le \Vert Ax{\Vert }_2^2\le (1+\delta )\Vert x{\Vert }_2^2\text{ for all }x\in {\mathbb{R}}^m\text{ with }\Vert x{\Vert }_0\le s\}.$$

$$(1-ϵ)\Vert u-v{\Vert }_{{\ell }^2({\mathbb{R}}^N)}^2\le \Vert f(u)-f(v){\Vert }_{{\ell }^2({\mathbb{R}}^{k_Q})}^2\le (1+ϵ)\Vert u-v{\Vert }_{{\ell }^2({\mathbb{R}}^N)}^2,\forall u,v\in Q.$$

$$\theta \Vert x{\Vert }_2^2\le \Vert A_{T}x{\Vert }_2^2\le \omega \Vert x{\Vert }_2^2$$

$$(1-\delta )\Vert x{\Vert }_2^2\le \Vert A_{T}x{\Vert }_2^2\le (1+\delta )\Vert x{\Vert }_2^2,\forall T\subseteq \{1,\dots ,n\}\text{ with }|T|\ge \tilde{n},\forall x\in {\mathbb{R}}^m\text{ with }\Vert x{\Vert }_0\le s.$$

$$n=C(s+n-\tilde{n})\ln \left(\frac{m+n}{s+n-\tilde{n}}\right),$$

$$\mathbb{P}(X=1)=\mathbb{P}(X=-1)=\frac{1}{2}.$$

$$\mathbb{E}(e^{tX})\le e^{\frac{b^2{t}^2}{2}},\forall t\in \mathbb{R}.$$

$$⟨X,\alpha ⟩\sim \mathrm{Sub}(\Vert \alpha {\Vert }_2^2{b}^2).$$

$$\mathbb{P}(\Vert Ax{\Vert }_2^2-\Vert x{\Vert }_2^2>ϵ\Vert x{\Vert }_2^2)<\exp (-\kappa {ϵ}^{2}n),$$

$$\mathbb{P}(\Vert Ax{\Vert }_2^2-\Vert x{\Vert }_2^2<-ϵ\Vert x{\Vert }_2^2)<\exp (-\kappa {ϵ}^{2}n).$$

$$\mathbb{P}\left(\frac{1}{n}\Vert Ax{\Vert }_2^2-\Vert x{\Vert }_2^2>ϵ\Vert x{\Vert }_2^2\right)<\exp \left(-\left(\frac{{ϵ}^2}{4}-\frac{{ϵ}^3}{6}\right)n\right)$$

$$\mathbb{P}\left(\frac{1}{n}\Vert Ax{\Vert }_2^2-\Vert x{\Vert }_2^2<-ϵ\Vert x{\Vert }_2^2\right)<\exp \left(-\left(\frac{{ϵ}^2}{4}-\frac{{ϵ}^3}{6}\right)n\right)$$

$$\mathbb{P}(f(X)-\mathbb{E}(f(X))\ge t)\le \exp \left(-\frac{t^2}{5b^2}\right),$$

$$\mathbb{P}(f(X)-\mathbb{E}(f(X))\le -t)\le \exp \left(-\frac{t^2}{5b^2}\right).$$

$$\gamma (t)=X^{\prime }\cos \left(\frac{\pi }{2}t\right)+X\sin \left(\frac{\pi }{2}t\right),\forall t\in [0,1].$$

$${\gamma }^{\prime }(t)=\frac{\pi }{2}\left(-X^{\prime }\sin \left(\frac{\pi }{2}t\right)+X\cos \left(\frac{\pi }{2}t\right)\right)=:\frac{\pi }{2}Y(t).$$

$$f(X)-f(X^{\prime })=\frac{\pi }{2}{\int }_0^1⟨\nabla f(\gamma (t)),Y(t)⟩\mathrm{d}t.$$

$$⟨\nabla f(\gamma (t)),Y(t)⟩\sim \mathrm{Sub}(b^2),\forall t\in [0,1].$$

$$\mathbb{E}(\exp (\lambda (f(X)-f(X^{\prime }))))\le {\int }_0^1\mathbb{E}(\exp (\frac{\pi }{2}\lambda ⟨\nabla f(\gamma (t)),Y(t)⟩))\mathrm{d}t\le \exp \left(\frac{b^2{\lambda }^2{\pi }^2}{8}\right)$$

$$\mathbb{E}(\exp (\lambda f(X^{\prime })))\ge \exp (\mathbb{E}(\lambda f(X^{\prime })))=1,\forall \lambda \in \mathbb{R}.$$

$$\mathbb{E}(\exp (\lambda f(X)))\le \mathbb{E}(\exp (\lambda (f(X)-f(X^{\prime }))))\le \exp \left(\frac{b^2{\lambda }^2{\pi }^2}{8}\right),\forall \lambda \in \mathbb{R}.$$

$$\mathbb{P}(f(X)\ge t)$$

$$\mathbb{P}(f(X)\ge t)\le \exp \left(-\frac{t^2}{5b^2}\right).$$

$$\mathbb{E}\left(\sqrt{\frac{1}{k}\sum _{j=1}^k{y}_{(j)}^2}\right)\le \sqrt{2e{b}^2\ln \frac{en}{k}},\forall k=1,\dots ,n.$$

$$F_S:{\mathbb{R}}^n\to \mathbb{R},y\mapsto \sqrt{\sum _{j\in S}{y}_{(j)}^2}.$$

$$|F_S(x)-F_S(y){|}^2$$

$$\le \sum _{j\in S}(x_{(j)}^2+y_{(j)}^2)-2\sum _{j\in S}|x_{(j)}{y}_{(j)}|$$

$$=\sum _{j\in S}(|x_{(j)}|-|y_{(j)}|{)}^2$$

$$\le \sum _{j=1}^n(|x_{(j)}|-|y_{(j)}|{)}^2$$

$$=\sum _{j=1}^n(x_{(j)}^2+y_{(j)}^2)-2\sum _{j=1}^n|x_{(j)}{y}_{(j)}|$$

$$\le \sum _{j=1}^n(x_j^2+y_j^2)-2\sum _{j=1}^n|x_j{y}_j|(\text{by the rearrangement inequality})$$

$$\le \Vert x-y{\Vert }_2^2.$$

Full text

Research supported in part by NSERC Canada under Grant RGP 228051.

1. Introduction

1.1. Background on Compressed Sensing

Consider the following problem: In the sensor network, we transmit a signal through independent channels and the center hub receives observations from each channel for further analysis. In practice, it is typical that some channels fail to send the correct measurements, and thus we can only obtain the corrupted data. To deal with problems like this, methods of reconstructing a signal from sampling observations are of great interest to researchers in the fields of engineering. In general, it is impossible to recover a signal if there is nothing known about the signal or the measurement. However, with prior knowledge of the signal, it is possible to perform the recovery with negligible or even zero error. Over the past years, researchers have been refreshing their understandings on the relevancy and practicability on the prior assumptions for the input signal and the measurement, and have developed several powerful tools to study the signal recovery problem.

In signal processing, compressed sensing (CS) is a technique for signal recovery via solving certain linear systems. In compressed sensing without corruption, the general acquisition setting is represented as $y=Ax$, where $x\in\mathbb{R}^{m}$ is a signal, $A\in\mathbb{R}^{n\times m}$ is a sensing matrix ( here $\mathbb{R}^{n\times m}$ denotes the set of all $n\times m$ real matrices), and $y\in\mathbb{R}^{n}$ is the measurement. In general, it is extremely difficult to recover the signal $x$ if we know nothing about $x$ itself and the sensing matrix $A$. Around 2004, it was shown by Candés, Romberg and Tao in [7], and Donoho in [12], that if a signal satisfies certain sparsity condition plus additional assumptions on the sensing matrix, then the reconstruction can be accomplished. The ideas established in those papers provided the foundations of CS.

A key concept introduced in CS is the restricted isometry property (RIP), which gives a characterization of “the almost norm preserving property” of a matrix. This property was first introduced by Candés and Tao in [8].

Definition 1.1**.**

A matrix $A\in\mathbb{R}^{n\times m}$ satisfies the restricted isometry property of order $s$ with $s\in\{1,\dots,m\}$, if there exists $\delta\in[0,1)$ such that

[TABLE]

for all $x\in\mathbb{R}^{m}$ with $\|x\|_{0}\leq s$, where $\|x\|_{0}$ denotes the number of non-zero entries of $x$. Moreover we define the $s$-RIP constant of $A$ via

[TABLE]

It was shown by Candés and Tao in [8] that one can exactly recover a sparse signal $x$ with $\|x\|_{0}\leq s$, if the $s$-RIP constant of the sensing matrix $A$ satisfies certain condition. In practice, matrices which satisfy the RIP are usually generated from random variables. There are many types of random matrices which have the RIP with overwhelming probability. We list few examples below:

• •

Matrices with entries drawn from i.i.d. Gaussian random variables.

• •

Matrices with entries drawn from i.i.d. symmetric Bernoulli random variables.

• •

Matrices with rows drawn randomly from a discrete Fourier transform matrix.

• •

Matrices with entries drawn from i.i.d. sub-gaussian or sub-exponential random variables.

For more examples and details, see e.g. [1, 3, 7, 13, 24, 26].

One the other hand, the RIP is closely related to the well-known Johnson-Lindenstrauss(JL) lemma, which was first stated and proved by Johnson and Lindenstrauss in [17] under an abstract setting. Here we provide a specific version of the JL lemma, which is the one we often use in CS:

Lemma 1.2** (Johnson-Lindenstrauss lemma).**

([3, Lemma 4.1]) Let $\epsilon\in(0,1)$ and $N\in\mathbb{N}$. For every finite subset $Q$ of $\mathbb{R}^{N}$, let $k_{Q}\in\mathbb{N}$ be such that $k_{Q}>O(\epsilon^{-2}\ln(|Q|))$, where $|Q|$ denotes the cardinality of the set $Q$. Then there exists a Lipschitz function $f:\mathbb{R}^{N}\to\mathbb{R}^{k_{Q}}$ such that

[TABLE]

The JL lemma says that a discrete set of points in a high dimensional Euclidean space can be embedded into a low dimensional space, in a way such that the distances between points are nearly preserved. From [3, Theorem 5.2], one can see that the JL lemma implies the RIP.

1.2. Compressed Sensing with Corruptions

A natural generalization of CS is CS with corruptions. In this case, some elements of the measurement are corrupted. We formulate the model in this setting as follows: Given a signal $x\in\mathbb{R}^{m}$ with certain sparsity condition and a sensing matrix $A\in\mathbb{R}^{n\times m}$. We only receive $A_{T}x$ as the observation, where $T\subseteq\{1,\dots,n\}$ is unknown and $A_{T}$ denotes the sub-matrix by keeping the rows of $A$ with indices in $T$. In order to reconstruct the signal accurately and efficiently, it is important to verify whether $A_{T}$ satisfies the RIP. This leads to the concept of the strong restricted isometry property (SRIP), which plays a central role in the study of the robustness of a matrix.

Definition 1.3**.**

A matrix $A\in\mathbb{R}^{n\times m}$ is said to satisfy the strong restricted isometry property (SRIP) of order $s$ and level $[\theta,\omega,\beta]$ with $0<\theta\leq 1\leq\omega<2$ and $\beta\in[0,1)$ if

[TABLE]

holds for all $x\in\mathbb{R}^{m}$ with $\|x\|_{0}\leq s$ and all $T\subseteq\{1,\dots,n\}\text{ with }|T^{c}|\leq\beta n.$

The concept of SRIP was introduced in [9], and the authors used the terminology ”democratic” to describe that all measurement elements are equally important. To be more specific, an $n\times m$ matrix $B$ is said to be $(\tilde{n},s,\delta)$-democratic with $\tilde{n}\in\{1,\dots,n\}$, $s\in\{1,\dots,m\}$ and $0<\delta<1$ if

[TABLE]

The erasure robostness property for Gaussian random matrices was investigated in [9], and the authors made the comment that the same analysis can be performed on sub-gaussian random matrices. It’s proved in [9] that, an $n\times m$ Gaussian (sub-gaussian) random matrix is $(\tilde{n},s,\delta)$-democratic with high probability provided that

[TABLE]

where $C>0$ is an absolute constant. One can see that the constraint (1) is slightly unnatural, comparing with the constraint $n=O(s\ln(m/s))$ as in traditional CS. Moreover, from the analysis performed in [9], it’s unclear whether the proportion of rows which are allowed to be corrupted depends on the size of the sensing matrix or the dimension of the measurement vector. Later, the definition of the SRIP, which is Definition1.3, was introduced in [30] This definition is more preferred because it sticks to the ratio rather than the number of rows being corrupted, which makes more sense for models with huge amount of data. It’s proved in [30] that Gaussian random matrices have the SRIP of certain level with overwhelming probability. In [15], further results about the robustness property of Gaussian random matrices were proved, including the robust version of the JL lemma for Gaussian random matrices.

In this paper, we concentrate our effort on Bernoulli random matrices, as they are used quite often as sensing matrices in CS ([21, 25, 27, 31]). A Bernoulli random matrix is defined as the following:

Definition 1.4**.**

We say that a real random variable $X$ has the symmetric Bernoulli distribution, if $X$ only takes values $\pm 1$ with

[TABLE]

A Bernoulli random matrix is a random matrix with entries drawn from i.i.d. symmetric Bernoulli random variables. It was proved that a Bernoulli random matrix is singular with positive probability (see [28]). The study on the singularity of a Bernoulli random matrix is closely related to its erasure robustness property. From the fact that a Bernoulli matrix is singular with positive probability, one may expect an upper bound on the portion of rows which can be erased, such that the corrupted Bernoulli random matrix will still satisfy the RIP with high probability. In fact, it is not difficult to show that the portion of rows erased cannot reach $\frac{1}{2}$. However, it is still unkonwn whether or not $\frac{1}{2}$ is the optimal upper bound for the erasure ratio.

1.3. Contributions and the Structure of the Paper

In this paper, we will prove that a Bernoulli random matrix satisfies the SRIP of certain order and level. Our study relies on the fact that a Bernoulli random variable belongs to the sub-gaussian class, which will be discussed in the next section. This indeed makes our study to share certain ingredients with [9], but still several new elements are involved in this paper. The key ingredient to yield the main result in [9] is the classical concentration of measure property for sub-gaussian random matrices, which also plays an important role in our study. However, with the classical concentration of measure property along, one may not expect to do better than [9], particularly one cannot deduce the result on the erasure ratio. Moreover, we have illustrated in the previous subsection that an unnatural constraint (1) has to be imposed in order to yield the main result in [9]. Thus in order to do better, we need to use several new tools, including a Lipschitz type concentration inequality, some order statistical techniques, and classical results on Bernoulli random variables/matrices. It’s also interesting to note that, our main theorem tells that an $n\times m$ Bernoulli random matrix $\Phi$ satisfies the SRIP with high probability of order $s$ and of certain level, where we only impose the constraint $n=O(s\ln(m/s))$, just as in traditional CS.

The structure of this paper is organized as follows: In section 2, a brief review on symmetric Bernoulli random variables will be given. We will provide some concentration inequalities and order statistics results, which are essential and will be used later. In section 3, the erasure robustness property of Bernoulli random matrices will be concretely treated. We will focus on the following question: if no more than $\beta n$ rows can be erased from an $n\times m$ Bernoulli random matrix $\Phi$ with some $\beta\in(0,1)$, how large can $\beta$ be so that the corrupted matrix still has the RIP with high probability? We will provide a lower bound on the erasure ratio $\beta$. As a direct consequence, we will establish the SRIP and a robust version of the JL lemma for Bernoulli random matrices. Finally, a summary and further discussion will be given in Section 4.

2. On Symmetric Bernoulli Random Variables

In this section, we provide some necessary background on Bernoulli random variables.

2.1. Sub-gaussian Random Variables

One of the most important properties of a Bernoulli random variable is that, it belongs to the sub-gaussian class (the definition will be provided later). Many of the auxiliary results on Bernoulli random variables are actually developed from the sub-gaussian property. Therefore, it’s better for us to review some basics on sub-gaussian random variables. All definitions and results mentioned in this subsection are known. For further details related to sub-gaussian random variables, we refer the readers to [5].

Definition 2.1**.**

A real random variable $X$ is called $b$-sub-gaussian for some $b>0$ if

[TABLE]

Notation: $X\sim\operatorname{Sub}(b^{2})$.

With the above definition, it’s easy to see that if $X$ is a symmetric Bernoulli random variable, then $X\sim\operatorname{Sub}(1)$.

It is straight forward to verify the following properties of sub-gaussian variables:

Proposition 2.2**.**

1. (i)

If $X\sim\operatorname{Sub}(b^{2})$, then $X$ is centred. That is, $\mathbb{E}(X)=0$. 2. (ii)

If $X\sim\operatorname{Sub}(b^{2})$, then $\operatorname{Var}(X)\leq b^{2}$. 3. (iii)

If $X\sim\operatorname{Sub}(b^{2})$, then for every $c\in\mathbb{R}$, we have $cX\sim\operatorname{Sub}(c^{2}b^{2})$. 4. (iv)

If $X_{1}\sim\operatorname{Sub}(b_{1}^{2})$ and $X_{2}\sim\operatorname{Sub}(b_{2}^{2})$, then $X_{1}+X_{2}\sim\operatorname{Sub}((b_{1}+b_{2})^{2})$. Moreover, if $X_{1}$ and $X_{2}$ are independent, then $X_{1}+X_{2}\sim\operatorname{Sub}(b_{1}^{2}+b_{2}^{2})$. 5. (v)

If $X=(X_{1},X_{2},\dots,X_{N})$ is a random vector such that $X_{i}$’s are i.i.d. random variables with $X_{i}\sim\operatorname{Sub}(b^{2})$ for all $i=1,\dots,N$. Then for every $\alpha\in\mathbb{R}^{N}$, we have

[TABLE]

Sub-gaussian random variables can be characterized in several equivalent ways:

Theorem 2.3**.**

The following statements are equivalent for a real centred random variable $X$:

1. (i)

$X\sim\operatorname{Sub}(b^{2})$* for some $b>0$.* 2. (ii)

For all $\lambda>0$, there exists $c>0$ such that $\mathbb{P}(|X|>\lambda)\leq 2e^{-c\lambda^{2}}$. In fact we may choose $c=\frac{1}{2b^{2}}$. 3. (iii)

For every $\xi>1$, there exists $a>0$ depending on $\xi$ such that $\mathbb{E}(e^{aX^{2}})\leq\xi$. In fact, we can choose $a=\frac{\xi-1}{2b^{2}(\xi+1)}$.

2.2. The Concentration of Measure Phenomena

The concentration of measure phenomena is extremely important in random matrix theory. It has been intensively studied over the past few years. For detailed discussions on this topic, we refer the readers to [1, 2, 10, 11, 16, 18, 19] and references therein.

For us to establish the SRIP for Bernoulli random matrices, we need two concentration inequalities for sub-gaussian random variables. The first inequality is the classical concentration of measure inequality for sub-gaussian random matrices, which was proved in [2, 11]:

Theorem 2.4** (Concentration of measure for sub-gaussian matrices).**

Suppose that $A\in\mathbb{R}^{n\times m}$ is a random matrix whose entries are drawn from i.i.d. $b$-sub-gaussian random variables for some $b>0$ and has variance $\frac{1}{n}$. Then for all $x\in\mathbb{R}^{m}$, we have:

1. (i)

$\mathbb{E}(\|Ax\|_{2}^{2})=\|x\|_{2}^{2}$. 2. (ii)

There exists $\kappa>0$ (depending on the sub-gaussian random variable used to generate $A$) such that for every $\epsilon\in(0,1)$, we have

[TABLE]

[TABLE]

Remark 2.5**.**

As a special case of Theorem2.4, it was proved in [1] that if $A\in\mathbb{R}^{n\times m}$ has entries drawn from i.i.d. standard normal or symmetric Bernoulli random variables, then we have

[TABLE]

and

[TABLE]

for all $x\in\mathbb{R}^{m}$ and $\epsilon\in(0,1)$. Thus in this case we may take $\kappa=\frac{1}{12}$.

Another inequality we need is a Lipschitz type concentration inequality for sub-gaussian random variables. This inequality for Gaussian random variables was established in [16, Proposition 2.3]. With only a minor modification of the proof of the Gaussian case, we can extend the result to sub-gaussian random variables. For the sake of convenience, we provide the result and its proof here.

Theorem 2.6** (Lipschitz concentration inequality for sub-gaussian distributions).**

Let $X=(X_{1},\dots,X_{d})$ has real i.i.d. entries with $X_{i}\sim\operatorname{Sub}(b^{2})$ for all $i=1,\dots,N$. Let $f:\mathbb{R}^{d}\to\mathbb{R}$ be a $1$-Lipschitz function. That is, $|f(x)-f(y)|\leq\|x-y\|_{2}$ for all $x,y\in\mathbb{R}^{d}$. Then for all $t>0$, we have

[TABLE]

[TABLE]

Proof.

By Rademacher’s theorem, a Lipschitz function is differentiable almost everywhere. So it suffices to prove the result for every differentiable function $f$, and the general case follows from a standard approximation argument. As $f$ has Lipschitz constant 1, we have $\|\nabla f\|_{2}\leq 1$. Without loss of generality, assume $\mathbb{E}(f(X))=0$. Let $X^{\prime}=(X^{\prime}_{1},\dots,X^{\prime}_{d})$ be an independent copy of $X$. That is, $X^{\prime}$ and $X$ have the same (joint) distribution and are indenepdent. Let $\gamma:[0,1]\to\mathbb{R}^{d}$ be a smooth path connecting $X$ and $X^{\prime}$ with

[TABLE]

Then

[TABLE]

By item (v) of Proposition2.2, $Y$ has i.i.d. components with $Y_{i}\sim\operatorname{Sub}(b^{2})$ for all $i=1,\dots,d$. Moreover, by the fundamental theorem of line integral

[TABLE]

As $\|\nabla f\|_{2}\leq 1$, and $Y$ has i.i.d. $\operatorname{Sub}(b^{2})$ entries, it follows that

[TABLE]

Thus by Jensen’s inequality, Fubini’s theorem and item (iii) of Proposition2.2, we have

[TABLE]

for all $\lambda\in\mathbb{R}$. As $X$ and $X^{\prime}$ are i.i.d. copies, we have $\mathbb{E}(f(X))=\mathbb{E}(f(X^{\prime}))=0$. Thus Jensen’s inequality yields

[TABLE]

Therefore

[TABLE]

Thus for all $\lambda,t>0$, we have

[TABLE]

By setting $\lambda=\frac{4t}{b^{2}\pi^{2}}\left(1+\sqrt{1-\frac{\pi^{2}}{10}}\right)$, we have

[TABLE]

This proves (3), and (4) can be proved similarly. ∎

A direct consequence of Theorem2.4 is the following corollary:

Corollary 2.7**.**

Suppose that $y_{1},\dots,y_{n}$ are i.i.d. random variables with $y_{i}\sim\operatorname{Sub}(b^{2})$ for $i=1,\dots,n$. Let $y_{(1)},\dots,y_{(n)}$ be the non-increasing rearrangements of $y_{i}$’s in magnitudes, i.e., $|y_{(1)}|\geq\dots\geq|y_{(n)}|$. Then

[TABLE]

Proof.

Let $S\subseteq\{1,\dots,n\}$. Define

[TABLE]

Note that for $x,y\in\mathbb{R}^{n}$:

[TABLE]

Thus $F_{S}$ is $1$-Lipschitz. For all $t>0$, it follows from Theorem2.4 that

[TABLE]

Let $\xi>1$ and choose $t=\frac{\xi-1}{2b^{2}(\xi+1)}$. It follows that:

[TABLE]

By taking logarithms on both sides and using Jensen’s inequality, we have

[TABLE]

Setting $\xi=e$ yields the result.∎

2.3. Linear Combinations of Symmetric Bernoulli Random Variables

To study the SRIP of Bernoulli random matrices, it is inevitable to encounter linear combinations of symmetric Bernoulli random variables. We first recall the famous Khinchine’s inequality:

Theorem 2.8** (Khinchine’s inequality).**

Let $X_{1},\dots,X_{m}$ i.i.d. symmetric Bernoulli random variables. Then for every $p\in(0,\infty)$, there exist $a(p),b(p)>0$ depending only on $p$ so that

[TABLE]

The optimal $a(p)$ and $b(p)$ are called the Khinchine’s constants.

The exact values of the Khinchine’s constants were given by Haagerup in [14] as follows:

Theorem 2.9**.**

The optimal $a(p)$ and $b(p)$ in Khinchine’s inequality are given by

[TABLE]

and

[TABLE]

where $p_{0}\in(0,2)$ satisfies $\Gamma\left(\frac{p_{0}+1}{2}\right)=\frac{\sqrt{\pi}}{2}$.

With Khinchine’s inequality and constants, and use the idea as in the proof of [4, Lemma 2.2], we can prove the following key lemma for Bernoulli random matrices.

Lemma 2.10**.**

Let $\Phi\in\mathbb{R}^{n\times m}$ be a Bernoulli random matrix. Then

[TABLE]

Proof.

Let $x_{0}\in\mathbb{S}^{m-1}$ and put $Y_{0}:=\|\Phi x_{0}\|_{2}^{2}$. For every $\mu\in\mathbb{R}$, define

[TABLE]

By Markov’s inequality, we have

[TABLE]

for all $\nu\in\mathbb{R}$. Recall that

[TABLE]

Denote $\phi_{ij}$ the entry of $\Phi$ in the $i-$th row and $j-$th colume. It follows that

[TABLE]

whenever $\mu-\frac{3}{2}\mu^{2}\geq 0$, where we have used the exact value of the Khinchine’s constant

[TABLE]

Therefore

[TABLE]

It follows that

[TABLE]

whenever $\mu-\frac{3}{2}\mu^{2}\geq 0$ and $\nu>0$. By setting $\mu=\frac{1}{3}$ and $\nu=\frac{qn}{3}$, the proof is complete.∎

3. Bernoulli Random Matrices under Arbitrary Erasure of Rows with a Given Portion of Corruption

In this section, we will study the erasure robustness property of Bernoulli random matrices. We will establish the SRIP and the robust version of Johnson-Lindenstrauss lemma for Bernoulli random matrices. Throughout this section:

• •

$\Phi\in\mathbb{R}^{n\times m}$ is a Bernoulli random matrix. As we are studying the erasure robusness property of the matrix, we should assume that our matrix has at least two rows ($n\geq 2$). In applications, usually the dimensions of matrices are huge.

• •

For $T\subseteq\{1,\dots n\}$, $\Phi_{T}$ denotes the sub-matrix of $\Phi$ by keeping rows with indices in $T$.

• •

For i.i.d. real random variables $y_{1},\dots,y_{n}$, we denote $y_{(1)},\dots,y_{(n)}$ the non-increasing rearrangements of $y_{i}$’s in magnitude. That is, $|y_{(1)}|\geq\dots\geq|y_{(n)}|$.

We introduce the following notations:

• •

Fix $x\in\mathbb{S}^{m-1}$. For $\beta\in[0,1)$ and $0\leq\theta\leq\omega\leq\infty$, define the following events:

[TABLE]

[TABLE]

• •

For $\beta\in[0,1]$, define

[TABLE]

• •

Fix $x\in\mathbb{S}^{m-1}$. For $\beta\in(0,1)$ and $\alpha>0$, define

[TABLE]

[TABLE]

[TABLE]

[TABLE]

3.1. Auxiliary Results

The following simple observation was proved in [15].

Lemma 3.1**.**

Fix $x\in\mathbb{S}^{m-1}$. For $0\leq\gamma\leq\beta<1$, we have

[TABLE]

and

[TABLE]

As a consequence, we have $\Omega_{[\theta,\omega],\beta}\subseteq\Omega_{[\theta,\omega],\gamma}$ for all $0\leq\theta\leq\omega\leq\infty$.

We now provide estimates for $\theta_{\beta}(\alpha),\omega_{\beta}(\alpha),\tilde{\theta}_{\beta}(\alpha),\tilde{\omega}_{\beta}(\alpha)$. Later we will use these estimates to establish our main results.

Lemma 3.2** (lower estimates for $\theta_{\beta}(\alpha)$ and $\tilde{\theta}_{\beta}(\alpha)$).**

For $\beta\in(0,1)$, choose $q_{\beta}\in\left(0,\frac{1}{2}\right)$ such that

[TABLE]

and let $\alpha\in\left(0,\ln\left((1-\beta)^{1-\beta}\beta^{\beta}\right)+\frac{q_{\beta}}{3}(1-\beta)\right)$. Denote $t_{\pm 1}\approx 0.0376$ the unique root of

[TABLE]

on $(0,\frac{1}{2})$. Then for each $\beta\in(0,t_{\pm 1})$, we have

[TABLE]

[TABLE]

Proof.

Fix $x\in\mathbb{S}^{m-1}$. For $\theta>0$ and $\beta\in\left(0,1\right)$, we have

[TABLE]

Let $\alpha>0$. To find a lower estimate for $\theta_{\beta}(\alpha)$, we need to determine the values of $\theta$ which satisfy

[TABLE]

Let $|T^{c}|=k\leq\beta n$ and $\gamma=k/n$. By Lemma2.10, we have

[TABLE]

Equivalently, we have

[TABLE]

Recall the Stirling’s approximation:

[TABLE]

It follows that for $k=1,\dots,n-1$, we have

[TABLE]

where in the last inequality we used the fact that $\frac{n}{(n-k)k}\leq 2$ whenever $n\geq 2$ and $1\leq k\leq n-1$. Now consider two cases:

• •

If $\lfloor\beta n\rfloor=0$, by Lemma2.10:

[TABLE]

By letting $\theta\leq\frac{1}{2}-3\alpha$ with $\alpha\in(0,\frac{1}{6})$, we have $\mathbb{P}(\Omega_{[\theta,\infty],\beta}^{c})<\exp(-\alpha n)$.

• •

If $\lfloor\beta n\rfloor\geq 1$, note that

[TABLE]

where the last equality follows from Lemma3.1 and $\beta_{n}=\frac{\lfloor\beta n\rfloor}{n}$. Let $k_{n}:=\beta_{n}n$, we have

[TABLE]

As (5) holds when $|T^{c}|=k_{n}$ and $\gamma=\beta_{n}$, it follows that for every $q\in\left(0,\frac{1}{2}\right)$:

[TABLE]

Now we want to bound $\mathbb{P}(\Omega_{[\theta,\infty],\beta}^{c})$ from above by $C\exp(-\alpha n)$ where $C$ is a positive constant independent of $n$. From what we have so far, it suffices to have $C=\frac{e}{\sqrt{2}\pi}$ and

[TABLE]

For every $c\in\left(0,\frac{1}{2}\right]$, define

[TABLE]

Then

[TABLE]

It is easy to see that $f^{\prime}_{c}(t)<0$ whenever $0<t<\frac{\exp\left(\frac{c}{3}\right)}{1+\exp\left(\frac{c}{3}\right)}$. Since $\frac{\exp\left(\frac{c}{3}\right)}{1+\exp\left(\frac{c}{3}\right)}\geq\frac{1}{2}$ whenever $0<c\leq\frac{1}{2}$, it follows that $f_{c}(t)$ is strictly decreasing on $\left(0,\frac{1}{2}\right)$, and

[TABLE]

On the other hand, note that

[TABLE]

It follows that $f_{c}$ has a unique root on $\left(0,\frac{1}{2}\right)$, call this root $t_{c}$. Thus we have

[TABLE]

Note that for $c^{\prime}\geq c$, we have

[TABLE]

Since $f_{c^{\prime}}(t)$ is strictly decreasing on $\left(0,\frac{1}{2}\right)$, it follows that $t_{c^{\prime}}\geq t_{c}$. That is, $t_{c}$ increases as $c$ increases. Define

[TABLE]

It follows that

[TABLE]

Whenever $\beta\in(0,t_{\pm 1})$, it’s not difficult to see that there exists $c_{\beta}\in\left(0,\frac{1}{2}\right)$ such that

[TABLE]

Indeed, $c_{\beta}=-3(\ln(1-\beta)+\frac{\beta}{1-\beta}\ln(\beta))$, see Figure 1 for an illustration. Choose $q_{\beta}\in(c_{\beta},\frac{1}{2})$, we have

[TABLE]

Now for $\alpha\in\left(0,(1-\beta)\ln(1-\beta)+\beta\ln(\beta)+\frac{q_{\beta}}{3}(1-\beta)\right)$, we have

[TABLE]

Therefore, if $\theta\leq\frac{1}{2}-3(1-\beta)^{-1}\left[\alpha-\ln\left((1-\beta)^{1-\beta}\beta^{\beta}\right)\right]$, it follows from (7) and (8) that

[TABLE]

Consequently, we have

[TABLE]

and

[TABLE]

As $f_{q_{\beta}}$ is decreasing on $(0,\frac{1}{2})$ and $\lim_{t\to 0^{+}}f_{q_{\beta}}(t)=\frac{q_{\beta}}{3}<\frac{1}{6}$, it follows that

[TABLE]

Thus $\alpha<\ln\left((1-\beta)^{1-\beta}\beta^{\beta}\right)+\frac{q_{\beta}}{3}(1-\beta)$ whenever $\beta\in(0,t_{\pm 1})$. This finishes the proof of the lower estimate for $\theta_{\beta}(\alpha)$.

To obtain the lower estimate for $\tilde{\theta}_{\beta}(\alpha)$, observe that

[TABLE]

where

[TABLE]

It follows that for $\theta>0$:

[TABLE]

Thus $\tilde{\theta}_{\beta}(\alpha)\geq(1-\beta)\theta_{\beta}(\alpha)$ and the lower estimate of $\tilde{\theta}_{\beta}(\alpha)$ follows immediately. ∎

The upper estimates for $\theta_{\beta}(\alpha)$ and $\tilde{\theta}_{\beta}(\alpha)$ can be deduced by applying the same argument as in the proof of [15, Lemma 4.1] with only a little modification.

Lemma 3.3** (upper estimates for $\theta_{\beta}(\alpha)$ and $\tilde{\theta}_{\beta}(\alpha)$).**

For $\beta\in(0,1)$ and $\alpha\in(0,1)$, we have $\theta_{\beta}(\alpha)\leq 1$ and $\tilde{\theta}_{\beta}(\alpha)\leq 1-\beta$.

Proof.

Fix $x\in\mathbb{S}^{m-1}$, let $y=Ax$. Set $\beta_{n}=\frac{\lfloor\beta n\rfloor}{n}$ and $k_{n}:=\beta_{n}n$. For every $\theta>0$, we have

[TABLE]

Note that

[TABLE]

If $\delta=\sqrt{\theta}-1>0$, we have

[TABLE]

If $\mathbb{P}(\Omega_{[\theta,\infty],\beta})\geq 1-e^{-\alpha n}$, we will have $1-e^{-\alpha n}\leq\exp\left(-\frac{\delta^{2}}{5}(1-\beta_{n})n\right)$ for all $n\geq 2$. By letting $n\to\infty$, we have $1\leq 0$, which is impossible. Hence we must have $\delta<0$, i.e. $\theta\leq 1$, and thus $\theta_{\beta}(\alpha)\leq 1$. Finally, $\tilde{\theta}_{\beta}(\alpha)\leq 1-\beta$ follows immediately from

[TABLE]

∎

Next, we estimate $\omega_{\beta}(\alpha)$ and $\tilde{\omega}_{\beta}(\alpha)$. The following result is a generalization of the corresponding result for the Gaussian case. One can verify that the result actually holds as long as $\Phi$ has entries drawn from i.i.d. sub-gaussian random variables.

Theorem 3.4** (Estimates of $\omega_{\beta}(\alpha)$ and $\tilde{\omega}_{\beta}(\alpha)$).**

For $\beta\in(0,1)$ and $\alpha>0$, we have

[TABLE]

[TABLE]

Proof.

Fix $x\in\mathbb{S}^{m-1}$, let $y=\Phi x$. Set $\beta_{n}=\frac{\lfloor\beta n\rfloor}{n}$ and $k_{n}:=\beta_{n}n$.

First we estimate $\tilde{\omega}_{\beta}(\alpha)$. For $\omega>0$, by definition:

[TABLE]

We claim that $\tilde{\omega}_{\beta}(\alpha)\geq 1$. Assume not, then there exists $\omega<1$ such that

[TABLE]

On the other hand, from the concentration inequality for Bernoulli random matrices (Theorem2.4 and Remark2.5), we have

[TABLE]

Hence we have $1-\exp(-\alpha n)<\exp\left(-\frac{1}{12}(1-\omega)^{2}n\right)$ for all $n\geq 2$. By letting $n\to\infty$, we have $1<0$, which is a contradiction. Therefore $\tilde{\omega}_{\beta}(\alpha)\geq 1$.

For the upper estimate for $\tilde{\omega}_{\beta}(\alpha)$:

[TABLE]

where $\varepsilon=\sqrt{12\alpha}$. Thus $\tilde{\omega}_{\beta}(\alpha)\leq 1+\sqrt{12\alpha}.$ This finishes the estimation for $\tilde{\omega}_{\beta}(\alpha)$.

Next, we estimate $\omega_{\beta}(\alpha)$. Note that for every $\omega\geq 0$, we have

[TABLE]

Since a Bernoulli random variable is sub-gaussian with $b=1$, Corollary2.7 yields

[TABLE]

Therefore

[TABLE]

provided that

[TABLE]

By $0\leq\beta_{n}\leq\beta<1$, it follows that $\mathbb{P}(\Omega_{[0,\omega],\beta})\geq 1-\exp(-\alpha n)$ if

[TABLE]

Therefore

[TABLE]

This proves the upper estimate for $\omega_{\beta}(\alpha)$. For the lower estimate of $\omega_{\beta}(\alpha)$, note that if $k_{n}=0$, then we have $\Omega_{[0,\omega],\beta}=\tilde{\Omega}_{[0,\omega],\beta}$, so $\omega_{\beta}(\alpha)\geq 1.$ If $k_{n}>0$, then $\beta_{n}>\frac{\beta}{2}$ and

[TABLE]

It follows that for $\omega\geq 0$:

[TABLE]

Hence

[TABLE]

This finishes the estimation for $\omega_{\beta}(\alpha)$, and the proof is now complete.∎

The last supporting result we need is the following well-known result (see e.g. [20]) on approximating the unit sphere with its finite sets.

Lemma 3.5**.**

Let $S\subseteq\{1,\dots,m\}$ with $|S|=s$. Set

[TABLE]

Then for any $\epsilon>0$ there exists an $\epsilon$-net $Q_{S,\epsilon}\subseteq\mathbb{S}^{m-1}_{S}$ satisfying

• •

$\mathbb{S}^{m-1}_{S}\subseteq\bigcup_{q\in Q_{S,\epsilon}}B_{\frac{\epsilon}{8}}(q)$, where $B_{\frac{\epsilon}{8}}(q)=\{v\in\mathbb{R}^{m}:\|v-q\|_{2}<\frac{\epsilon}{8}\}$ .

• •

$|Q_{S,\epsilon}|\leq\left(\frac{24}{\epsilon}\right)^{s}$.

3.2. Main Results

With the auxiliary results derived in the previous subsection, we are at the stage to prove the SRIP of Bernoulli random matrices. In order to make the statement of the main theorem simple, we fix the following notations:

• •

For each $\beta\in(0,1)$, set

[TABLE]

for all $t\in(0,1)$.

• •

$t_{\pm 1}\approx 0.0376$ is the unique zero of the function

[TABLE]

on $(0,\frac{1}{2})$.

Theorem 3.6** (the strong restricted isometry property of Bernoulli random matrices).**

Let $\beta\in(0,t_{\pm 1})$ and choose $q_{\beta}\in(0,\frac{1}{2})$ such that $\alpha_{\beta}(q_{\beta})>0$. Let $s,m,n\in\mathbb{N}$, $\alpha\in(0,\min\{\alpha_{\beta}(q_{\beta}),\frac{1}{12}\})$ and $\epsilon\in(0,1)$ be such that

[TABLE]

Let $\Phi$ be an $n\times m$ Bernoulli random matrix. Then

[TABLE]

and

[TABLE]

where $\tilde{\theta}_{\epsilon,\beta},\tilde{\omega}_{\epsilon,\beta},\theta_{\epsilon,\beta},\omega_{\epsilon,\beta}$ are positive constants which only depend on $\beta$ and $\epsilon$.

Proof.

The proof is based on the the ideas from the proofs of [[30], Theorem 2.1] and [[3], Lemma 5.1].

Let $T\subseteq\{1,\dots,n\}$ be such that $|T^{c}|\leq\beta n$. Define

[TABLE]

[TABLE]

[TABLE]

By Lemma3.2 and 3.4, with probability at least $1-2e^{-\alpha n}$, we have

[TABLE]

for any $\epsilon\in(0,1)$. Let $S$, $\mathbb{S}^{m-1}_{S}$ and $Q_{S,\epsilon}$ be the same as in Lemma3.5. Define

[TABLE]

For every $u\in\mathbb{S}^{m-1}_{S}$, there exists $v_{u}\in Q_{S,\epsilon}$ such that $\|u-v_{u}\|_{2}\leq\frac{\epsilon}{8}$. Hence

[TABLE]

By the definition of $d$, we have $d\leq\sqrt{\tilde{\omega}(1+\epsilon)}+\frac{d\epsilon}{8}$, which implies that $d\leq\sqrt{\tilde{\omega}(1+2\epsilon)}.$

On the other hand, Lemma3.2 and 3.4 yield

[TABLE]

Choose $\epsilon>0$ small enough such that $\sqrt{\tilde{\theta}}-\frac{\epsilon}{8}\sqrt{\tilde{\omega}}>0$. Set

[TABLE]

[TABLE]

One can see that with probability at least $1-2\left(\frac{24}{\epsilon}\right)^{s}\exp(-\alpha n)$, we have

[TABLE]

As there are $\binom{m}{s}$ subsets of $\{1,\dots,m\}$ with cardinality $s$, then by the union bound argument and Stirling’s approximation $\binom{m}{s}\leq\left(\frac{em}{s}\right)^{s}$, we see that

[TABLE]

and similarly

[TABLE]

provided that $s\ln\frac{24em}{\epsilon s}<\alpha n-\ln 2$. Thus the proof is complete.∎

Another result which we can establish is the following robust version of the Johnson-Lindenstrauss lemma:

Theorem 3.7** (robust Johnson-Lindenstrauss lemma for Bernoulli random matrices).**

Let $\beta\in(0,t_{\pm 1})$ and choose $q_{\beta}\in(0,\frac{1}{2})$ such that $\alpha_{\beta}(q_{\beta})>0$. Let $N,m,n\in\mathbb{N}$ and $\alpha\in(0,\min\{\alpha_{\beta}(q_{\beta}),\frac{1}{12}\})$ be such that

[TABLE]

Let $\Phi$ be an $n\times m$ Bernoulli random matrix. Then for every $N$-point subset $\{p_{1},\dots,p_{N}\}$ of $\mathbb{R}^{m}$, we have

[TABLE]

[TABLE]

where $\tilde{\theta}$, $\tilde{\omega}$ and $\omega$ are defined as (11), (12) and (13).

Proof.

By Lemma3.2 and3.4, we have

[TABLE]

and

[TABLE]

for every pair $(j,k)$ with $1\leq j,k\leq N$ and $j\neq k$. As there are $\binom{N}{2}=\frac{N(N-1)}{2}$ pairs $(p_{j},p_{k})$ with $j\neq k$, then by the union bound argument and the assumption $n>\alpha^{-1}\ln[N(N-1)]$, the result follows.∎

4. Summary and Discussion

In this paper, we have studied the erasure robustness property of Bernoulli random matrices. We’ve proved the SRIP and a robust version of the Johnson-Lindenstrauss lemma for Bernoulli random matrices. From our analysis, we see that if the portion of rows erasured is smaller than the number $t_{\pm 1}$ which was provided in Theorem3.6, then a Bernoulli random matrix will satisfy the SRIP with high probability. One natural question to ask is that, what happens if the erasure ratio is above $t_{\pm 1}$? Moreover, we are interested in the optimal upper bound of the erasure ratio for the SRIP to hold. We know that an upper bound is $\frac{1}{2}$, but whether or not it’s optimal is unknown. It seems that to fully solve this problem, significantly new ideas and techniques are required.

Acknowledgements

I would like to express my sincere gratitude to my supervisor Professor Bin Han, for his patient help and encouragements.