Uniform Recovery Bounds for Structured Random Matrices in Corrupted Compressed Sensing

TL;DR

This paper establishes uniform recovery guarantees for structured random matrices in corrupted compressed sensing, enabling exact and stable sparse signal recovery despite measurement corruption and noise.

Contribution

It proves new uniform recovery bounds for two classes of structured sensing matrices under corruption, extending compressed sensing theory.

Findings

Recovery guaranteed for partial random circulant matrices with high probability.

Stable recovery of sparse signals with Fourier matrices under certain sparsity conditions.

Theoretical results supported by simulation experiments.

Abstract

We study the problem of recovering an $s$-sparse signal $\mathbf{x}^{\star}\in\mathbb{C}^n$ from corrupted measurements $\mathbf{y} = \mathbf{A}\mathbf{x}^{\star}+\mathbf{z}^{\star}+\mathbf{w}$, where $\mathbf{z}^{\star}\in\mathbb{C}^m$ is a $k$-sparse corruption vector whose nonzero entries may be arbitrarily large and $\mathbf{w}\in\mathbb{C}^m$ is a dense noise with bounded energy. The aim is to exactly and stably recover the sparse signal with tractable optimization programs. In this paper, we prove the uniform recovery guarantee of this problem for two classes of structured sensing matrices. The first class can be expressed as the product of a unit-norm tight frame (UTF), a random diagonal matrix and a bounded columnwise orthonormal matrix (e.g., partial random circulant matrix). When the UTF is bounded (i.e. $\mu(\mathbf{U})\sim1/\sqrt{m}$), we prove that with high probability, one can recover an $s$-sparse signal exactly and stably by $l_1$ minimization programs even if the measurements are corrupted by a sparse vector, provided $m = \mathcal{O}(s \log^2 s \log^2 n)$ and the sparsity level $k$ of the corruption is a constant fraction of the total number of measurements. The second class considers randomly sub-sampled orthogonal matrix (e.g., random Fourier matrix). We prove the uniform recovery guarantee provided that the corruption is sparse on certain sparsifying domain. Numerous simulation results are also presented to verify and complement the theoretical results.

Equations

$$\mathbf{y}=\mathbf{A}{\mathbf{x}}^{\star }+\mathbf{w},$$

$$\mathbf{y}=\mathbf{A}{\mathbf{x}}^{\star }+{\mathbf{z}}^{\star }+\mathbf{w},$$

$$\underset{\mathbf{x},\mathbf{z}}{\min }\Vert \mathbf{x}{\Vert }_1+\lambda \Vert \mathbf{z}{\Vert }_{1}s.t.\Vert \mathbf{y}-(\mathbf{Ax}+\mathbf{z}){\Vert }_2\le \epsilon .$$

$$\mathbf{y}=\mathbf{A}{\mathbf{x}}^{\star }+\mathbf{H}{\mathbf{z}}^{\star }+\mathbf{w},$$

$${\sigma }_s(\mathbf{a}{)}_p=\underset{\Vert \tilde{\mathbf{a}}{\Vert }_0\le s}{\inf }\Vert \mathbf{a}-\tilde{\mathbf{a}}{\Vert }_p,$$

$$(1-{\delta }_s)\Vert \mathbf{x}{\Vert }_2^2\le \Vert \mathbf{Ax}{\Vert }_2^2\le (1+{\delta }_s)\Vert \mathbf{x}{\Vert }_2^2,$$

$${\delta }_s=\underset{\mathbf{x}\in {\mathcal{D}}_{s,n}}{\sup }\left|\Vert \mathbf{Ax}{\Vert }_2^2-\Vert \mathbf{x}{\Vert }_2^2\right|,$$

$$(1-\delta )(\Vert \mathbf{x}{\Vert }_2^2+\Vert \mathbf{z}{\Vert }_2^2)\le {\Vert \mathbf{\Theta }\left[\begin{array}{c}\mathbf{x}\\ \mathbf{z}\end{array}\right]\Vert }_2^2\le (1+\delta )(\Vert \mathbf{x}{\Vert }_2^2+\Vert \mathbf{z}{\Vert }_2^2)$$

$${\delta }_{2s,2k}<\frac{1}{\sqrt{1+{\left(\frac{1}{2\sqrt{2}}+\sqrt{\eta }\right)}^2}}$$

$$\displaystyle\|\hat{\mathbf{x}}-\mathbf{x}^{\star}\|_{1}+\|\hat{\mathbf{z}}-\mathbf{z}^{\star}\|_{1}\$$

$$+c_2\sqrt{s+{\lambda }^{2}k}\epsilon$$

$$\displaystyle\|\hat{\mathbf{x}}-\mathbf{x}^{\star}\|_{2}+\|\hat{\mathbf{z}}-\mathbf{z}^{\star}\|_{2}\$$

$$+c_4\left(1+{\eta }^{1/4}\right)\epsilon ,$$

$$m$$

$${\delta }_{s,k}=\underset{(\mathbf{x},\mathbf{z})\in \mathcal{T}}{\sup }\left|{\Vert \mathbf{\Theta }\left[\begin{array}{c}\mathbf{x}\\ \mathbf{z}\end{array}\right]\Vert }_2^2-\Vert \mathbf{x}{\Vert }_2^2-\Vert \mathbf{z}{\Vert }_2^2\right|,$$

$${\delta }_{s,k}$$

$$\le \underset{{\delta }_1}{\underbrace{\underset{(\mathbf{x},\mathbf{z})\in \mathcal{T}}{\sup }\left|\Vert \mathbf{Ax}{\Vert }_2^2-\Vert \mathbf{x}{\Vert }_2^2\right|}}+\underset{{\delta }_2}{\underbrace{2\underset{(\mathbf{x},\mathbf{z})\in \mathcal{T}}{\sup }|⟨\mathbf{Ax},\mathbf{z}⟩|}}$$

$${\delta }_1$$

$$m\ge 4c_1{\delta }^{-2}s\tilde{n}{\mu }^2(\tilde{\mathbf{B}})({\log }^{2}s{\log }^2\tilde{n}),$$

$${\delta }_2$$

$$=2\underset{(\mathbf{x},\mathbf{z})\in \mathcal{T}}{\sup }|{\mathbf{z}}^{\ast }\mathbf{U}\text{diag}(\tilde{\mathbf{B}}\mathbf{x})\mathbf{\xi }|=2\underset{\mathbf{v}\in {\mathcal{A}}_{\mathbf{v}}}{\sup }|⟨\mathbf{v},\mathbf{\xi }⟩|,$$

$${\mathcal{A}}_{\mathbf{v}}:=\{\mathbf{v}:\Vert \mathbf{x}{\Vert }_2^2+\Vert \mathbf{z}{\Vert }_2^2=1,\Vert \mathbf{x}{\Vert }_0\le s,\Vert \mathbf{z}{\Vert }_0\le k\}.$$

$$m$$

$$1-{\tilde{n}}^{-(\log \tilde{n}{\log }^{2}s)}-\exp (-c{\log }^{2}s{\log }^2\tilde{n})$$

$$=1-{\tilde{n}}^{-(\log \tilde{n}{\log }^{2}s)}-{\tilde{n}}^{-c{\log }^{2}s\log \tilde{n}}$$

$$=1-2{\tilde{n}}^{-{\log }^{2}s\log \tilde{n}}$$

$$\mathbf{A}=\frac{1}{\sqrt{m}}{\mathbf{R}}_{\Omega }{\mathbf{C}}_{\mathbf{ϵ}}$$

$$\mathbf{y}=\mathbf{A}{\mathbf{x}}^{\star }+\mathbf{H}{\mathbf{z}}^{\star }+\mathbf{w}.$$

$${\mathbf{H}}^{\ast }\mathbf{y}={\mathbf{H}}^{\ast }\mathbf{A}{\mathbf{x}}^{\star }+{\mathbf{z}}^{\star }+{\mathbf{H}}^{\ast }\mathbf{w}.$$

$$\mathbf{y}=\mathbf{A}{\mathbf{x}}^{\star }+\mathbf{H}{\mathbf{z}}^{\star }+\mathbf{w},$$

$$m$$

$$\mathbf{y}=\mathbf{A}{\mathbf{x}}^{\star }+\mathbf{H}{\mathbf{z}}^{\star },$$

Full text

I Introduction

The theory of compressed sensing has been widely studied and applied in various promising applications over the recent years [1, 2, 3, 4, 5]. It provides an efficient way to recover a sparse signal from a relatively small number of measurements. Specifically, an $s$-sparse signal $\mathbf{x}^{\star}$ is measured through

[TABLE]

where $\mathbf{A}\in\mathbb{C}^{m\times n}$ is referred to as the sensing matrix, $\mathbf{y}\in\mathbb{C}^{m}$ is the measurement vector and $\mathbf{w}\in\mathbb{C}^{m}$ represents the noise vector with the noise level $\|\mathbf{w}\|_{2}\leq\varepsilon$. It has been shown that if $\mathbf{A}$ satisfies the restricted isometry property (RIP) and $\varepsilon$ is small, the recovered signal $\hat{\mathbf{x}}$ obtained by $l_{1}$ norm minimization is close to the true $\mathbf{x}^{\star}$, i.e. $\|\hat{\mathbf{x}}-\mathbf{x}^{\star}\|\leq C\varepsilon$ with $C$ being a small numerical constant. Many types of sensing matrices have been proven to satisfy the RIP condition. For example, random Gaussian/Bernoulli matrices satisfy the RIP with high probability if $m\geq\mathcal{O}(s\log(n/s))$ [3, 1], whereas structured sensing matrices consisting of either randomly subsampled orthonormal matrix [6] or modulated unit-norm tight frames [7] have the RIP with high probability when $m$ is about $\mathcal{O}(s\log^{4}n)$111Recent works [8][9] for subsampled Fourier matrices show that the factor $\log^{4}n$ can be reduced to $\log^{3}n$..

This standard compressed sensing problem has been generalized to cope with the recovery of sparse signals when some unknown entries of the measurement vector $\mathbf{y}$ are severely corrupted. Mathematically, we have

[TABLE]

where $\mathbf{z}^{\star}\in\mathbb{C}^{m}$ is an unknown sparse vector. To reconstruct $\mathbf{x}^{\star}$ from the measurement vector $\mathbf{y}$, the following penalized $l_{1}$ norm minimization has been proposed:

[TABLE]

In [10], it was shown that random Gaussian matrices can provide uniform recovery guarantees to this problem (3). In other words, a single random draw of a Gaussian matrix $\mathbf{A}$ is able to stably recover all $s$-sparse signals $\mathbf{x}^{\star}$ and all $k$-sparse corruptions $\mathbf{z}^{\star}$ simultaneously with high probability. On the other hand, for structured sensing matrices, the nonuniform recovery guarantees222A nonuniform recovery result only states that a fixed pair of sparse signal and sparse corruption can be recovered with high probability using a random draw of the matrix. Sometimes, the signs of the non-zero coefficients of the sparse vector (and corruption) can be chosen at random to further simplify arguments. Uniform recovery is stronger than nonuniform recovery. (see [5, Chapter 9.2][6, Section 3.1] for more details.) can be proved for randomly subsampled orthonormal matrix [11] and its generalized model - bounded orthonormal systems333See [12] for the construction of the generalized model. [10]. Very recently, the uniform recovery guarantee for bounded orthonormal systems is shown in [13].

In this paper, we prove the uniform recovery guarantee for two different corrupted sensing models. In the first model, the measurement matrix is based on randomly modulated unit-norm frames [7] and the corruption is sparse on the identity basis. It is noted that the measurement matrix in the first model does not consist of a random subsampling operator, e.g., the partial random circulant matrix [14]. For the second model, we consider

[TABLE]

where $\mathbf{A}$ represents a randomly subsampled orthonormal matrix, and the corruption $\mathbf{H}\mathbf{z}^{\star}$ is assumed to be sparse on certain bounded domain (e.g., a discrete Fourier transform (DFT) matrix). Our results imply that many structured sensing matrices can be employed in the corrupted sensing model to ensure the exact and stable recovery of both $\mathbf{x}^{\star}$ and $\mathbf{z}^{\star}$, even when the sparsity of the corruption is up to a constant fraction of the total number of measurements. Thanks to the uniform recovery guarantee, our results can address the adversarial setting, which means that exact and stable recovery is still guaranteed even when $\mathbf{x}^{\star}$, $\mathbf{z}^{\star}$ and $\mathbf{w}$ are selected given knowledge of the sensing matrix $\mathbf{A}$. In addition, our analysis results are also applicable to demonstrate the recovery guarantee when the corrupted sensing problem is solved via nonconvex optimization.

I-A Potential Applications

The problem of recovering sparse signal $\mathbf{x}^{\star}$ and sparse corruption $\mathbf{z}^{\star}$ from the measurement vector $\mathbf{y}$ arise from many applications, where the compressed measurements may be corrupted by impulse noise.

For example, in a sensor network, each sensor node measures the same signal $\mathbf{x}^{\star}$ independently before sending the outcome to the center hub for analysis. In this setting, each sensor makes the measurement $\langle\mathbf{a}_{i},\mathbf{x}^{\star}\rangle$, and the resultant measurement vector is $\mathbf{A}\mathbf{x}^{\star}$ by arranging each $\mathbf{a}_{i}$ as the rows of $\mathbf{A}$ [15, 11]. However, in practice, some sensor readings can be anomalous from the rest. These outliers could be caused by individually malfunctioned sensors, or due to some unusual phenomena or event happening in certain areas of the network [16][17]. This anomaly effect can be modeled by a sparse vector $\mathbf{z}^{\star}$. Mathematically, we have $\mathbf{y}=\mathbf{A}\mathbf{x}^{\star}+\mathbf{z}^{\star}+\mathbf{w}$, where $\mathbf{z}^{\star}$ represents the outlier regions and $\mathbf{w}$ stands for possible small noise in the data transmission. Our results make it possible to recover both the underlying signal and detect the outlier regions simultaneously, which could be very useful for network monitoring.

Another application of sparse signal recovery from sparsely corrupted measurements is error correction in joint source-channel coding. In [18, 19, 11], compressed sensing has been exploited as a joint source-channel coding strategy for its efficient encoding and robust error correcting performance. For a signal $\mathbf{f}$ that is sparse in the domain $\boldsymbol{\Psi}$, i.e., $\mathbf{f}=\boldsymbol{\Psi}\mathbf{x}^{\star}$, it can be encoded by a linear projection $\mathbf{y}=\boldsymbol{\Phi}\mathbf{f}=\mathbf{A}\mathbf{x}^{\star}$ with $\mathbf{A}=\boldsymbol{\Phi}\boldsymbol{\Psi}$. Existing works have investigated the situations where the encoded signal $\mathbf{y}$ is sent through either an erasure channel [18] or a gross error channel [19, 11]. Our results can not only be applied in these scenarios, but also provide a new design on the encoding matrix with uniform recovery guarantee.

In some scenarios, the measurement noise may be sparse or compressible in some sparsifying basis. One example is the recovery of video or audio signal that are corrupted by narrow-band interference (NBI) due to improper designed equipment [20, 21]. Electric hum as a typical impairment is sparse in the Fourier basis. Another example is the application of compressed sensing to reduce the number of samples in convolution systems with deterministic sequences (e.g., m-sequence, Golay sequence). Such convolution systems are widely used in communications, ultrasound and radar [22, 23]. In practice, the measurements may be affected by frequency domain interference or multi-tone jamming [24]. For instance, in CS-based OFDM channel estimation [25, 26, 27, 28], suppose $\mathbf{x}$ is the channel response and that the pilot sequence $\mathbf{g}$ is constructed from Golay sequences, the time-domain received signal can be represented as [28], $\mathbf{y}=\sqrt{\frac{n}{m}}\mathbf{R}_{\Omega^{\prime}}\mathbf{F}^{*}\text{diag}(\mathbf{g})\mathbf{F}\mathbf{x}+\mathbf{w}$, where $\mathbf{R}_{\Omega^{\prime}}$ is a random subsampling operator and $\mathbf{F}$ denotes the DFT matrix. The recovery performance can be guaranteed by noticing that the sensing model is a subsampled version of the orthonormal matrix $\mathbf{F}^{*}\text{diag}(\mathbf{g})\mathbf{F}$. However, in OFDM-based powerline communications, the NBI due to intended or unintended narrow-band signals can severely contaminate the transmitted OFDM signal. The time-domain NBI vector is sparse in the Fourier basis [29, 30]. Our results cover these settings, and therefore, provide a CS-based method to jointly estimate the signal of interest and the NBI.

I-B Notations and Organization of the paper

For an $n$-element vector $\mathbf{a}$, we denote by $a_{i}$, ($i\in[n]=\{0,...,n-1\}$), the $i$-th element of this vector. We represent a sequence of vectors by $\mathbf{a}_{0},...,\mathbf{a}_{n-1}$ and a column vector with $q$ ones by $\mathbf{1}_{q}$. The sparsity of a vector can be measured by its best $s$-term approximation error,

[TABLE]

where $\|\cdot\|_{p}$ is the standard $l_{p}$ norm on vectors. For a matrix $\mathbf{A}$, $A_{jk}$ denotes the element on its $j$-th row and $k$-th column. The vector obtained by taking the $j$-th row ($k$-th column) of $\mathbf{A}$ is represented by $\mathbf{A}_{(j,:)}$ ($\mathbf{A}_{(:,k)}$). We denote by $\mathbf{A}_{0},...,\mathbf{A}_{n-1}$ a sequence of matrices. $\mathbf{A}^{-1}$ and $\mathbf{A}^{*}$ represent the inverse and the conjugate transpose of $\mathbf{A}$. The Frobenius norm and the operator norm of matrix $\mathbf{A}$ are denoted by $\|\mathbf{A}\|_{F}=\sqrt{\text{tr}(\mathbf{A}^{*}\mathbf{A})}$ and $\|\mathbf{A}\|_{2\rightarrow 2}=\sup_{\|\mathbf{x}\|_{2}=1}\|\mathbf{A}\mathbf{x}\|_{2}$ respectively. We write $A\lesssim B$ if there is an absolute constant $c$ such that $A\leq cB$. We denote $A\sim B$ if $c_{1}A\leq B\leq c_{2}A$ for absolute constants $c_{1}$ and $c_{2}$.

The coherence $\mu(\mathbf{A})$ of an $\tilde{n}\times n$ matrix $\mathbf{A}$ describes the maximum magnitude of the elements of $\mathbf{A}$, i.e., $\mu(\mathbf{A})=\max_{\begin{subarray}{c}1\leq j\leq\tilde{n}\\ 1\leq k\leq n\end{subarray}}|\mathbf{A}_{jk}|$. For a unitary matrix $\boldsymbol{\Psi}\in\mathbb{C}^{n\times n}$, we have $\frac{1}{\sqrt{n}}\leq\mu(\boldsymbol{\Psi})\leq 1$.

The rest of the paper is organized as follows. We start by reviewing some key notions and results in compressed sensing in Section II. In Section III, we prove the uniform recovery guarantee for two classes of structured random matrices. In Section IV, we conduct a series of simulations to reinforce our theoretical results. Conclusion is given in Section V. We defer most of the proofs to the Appendices.

II Preliminaries

II-A RIP and structured sensing matrices

The restricted isometry property (RIP) is a sufficient condition that guarantees uniform and stable recovery of all $s$-sparse vectors via nonlinear optimization (e.g. $l_{1}$-minimization). For a matrix $\mathbf{A}\in\mathbb{C}^{m\times n}$ and $s<n$, the restricted isometry constant $\delta_{s}$ is defined as the smallest number such that

[TABLE]

holds for all $s$-sparse vectors $\mathbf{x}$. Alternatively, the restricted isometry constant of $\mathbf{A}$ can be written as

[TABLE]

where $\mathcal{D}_{s,n}=\{\mathbf{x}\in\mathbb{C}^{n}:\|\mathbf{x}\|_{2}\leq 1,\|\mathbf{x}\|_{0}\leq s\}$.

Among the many structured sensing matrices that satisfy the RIP, two classes have been found to be applicable in various scenarios. One is the randomly subsampled orthonormal systems [6], which encompass structured sensing matrices like partial random Fourier [2], convolutional CS [31, 28] and spread spectrum [32]. The other is the UDB framework which consists of a unit-norm tight frame (UTF), a random diagonal matrix and a bounded column-wise orthonormal matrix [7]. Popular sensing matrices under this framework include partial random circulant matrices [14], random demodulation [33], random probing [34] and compressive multiplexing [35].

II-B Recovery Condition

We review the definition of generalized RIP, which is useful to establish robustness and stability of the optimization algorithm.

Definition II.1**.**

[10, Definition 2.1]** For any matrix $\boldsymbol{\Theta}\in\mathbb{C}^{r\times(n+m)}$, it has the $(s,k)$-RIP with constant $\delta_{s,k}$ if $\delta_{s,k}$ is the smallest value of $\delta$ such that

[TABLE]

holds for any $\mathbf{x}\in\mathbb{C}^{n}$ with $\|\mathbf{x}\|_{0}\leq s$ and any $\mathbf{z}\in\mathbb{C}^{m}$ with $\|\mathbf{z}\|_{0}\leq k$.

Here, the generalized RIP is termed as the $(s,k)$-RIP for convenience. We note that the $(s,k)$-RIP is more stringent than the conventional RIP. In other words, the fact that a sensing matrix $\mathbf{A}$ satisfies the RIP does not mean that the associated matrix $\boldsymbol{\Theta}=[\mathbf{A},\ \mathbf{I}]$ would satisfy the $(s,k)$-RIP. The recovery performance of the penalized optimization (3) can be guaranteed by the following result.

Theorem II.2**.**

[13, Theorem 3.7]** Suppose $\mathbf{y}=\mathbf{A}\mathbf{x}^{\star}+\mathbf{z}^{\star}+\mathbf{w}$ and $\boldsymbol{\Theta}=[\mathbf{A},\ \mathbf{I}]\in\mathbb{C}^{m\times(n+m)}$ has the $(2s,2k)$-RIP constant $\delta_{2s,2k}$ satisfying

[TABLE]

with $\eta=\frac{s+\lambda^{2}k}{\min\{s,\lambda^{2}k\}}$. Then for $\mathbf{x}^{\star}\in\mathbb{C}^{n}$, $\mathbf{z}^{\star}\in\mathbb{C}^{m}$, and $\mathbf{w}\in\mathbb{C}^{m}$ with $\|\mathbf{w}\|_{2}\leq\varepsilon$, the solution $(\hat{\mathbf{x}},\hat{\mathbf{z}})$ to the penalized optimization problem (3) satisfies

[TABLE]

where the constants $c_{1}$, $c_{2}$, $c_{3}$, $c_{4}$ depend on $\delta_{2s,2k}$ only.

We note that similar theorem has been proven in [10] when both the signal and corruption are vectors with exact sparsity. The above result not only relaxes the requirement on the $(2s,2k)$-RIP constant, but also guarantees stable recovery of inexactly sparse signals and corruptions. Therefore, for either sparse or compressible signals and corruptions, the key to establish the recovery guarantee for a sensing matrix is to prove the $(s,k)$-RIP.

III Main Results

In this section, we prove the $(s,k)$-RIP for two classes of structured sensing matrices. This result can then be combined with Theorem II.2 to prove the recovery guarantee. In addition, the extension to the recovery via nonconvex optimization is presented. Last but not least, we compare the main theorems to existing literature where relevant.

III-A Randomly modulated unit-norm tight frames

We prove the uniform recovery guarantees for the class of structured sensing matrices that can be written as $\mathbf{A}=\mathbf{U}\mathbf{D}\widetilde{\mathbf{B}}$, where $\mathbf{U}\in\mathbb{C}^{m\times\tilde{n}}$ is a UTF with $\mu(\mathbf{U})\sim 1/\sqrt{m}$, $\mathbf{D}=\text{diag}(\boldsymbol{\xi})$ is a diagonal matrix with $\boldsymbol{\xi}$ being a length-$\tilde{n}$ random vector with independent, zero-mean, unit-variance, and $L$-subgaussian entries, and $\widetilde{\mathbf{B}}\in\mathbb{C}^{\tilde{n}\times n}$, $\tilde{n}\geq n$, represents a column-wise orthonormal matrix, i.e. $\widetilde{\mathbf{B}}^{*}\widetilde{\mathbf{B}}=\mathbf{I}$.

The following result presents a bound on the required number of measurements $m$ such that the corresponding matrix $\boldsymbol{\Theta}$ has the $(s,k)$-RIP constant satisfying $\delta_{s,k}\leq\delta$ for any $\delta\in(0,1)$.

Theorem III.1**.**

Suppose $\mathbf{y}=\mathbf{A}\mathbf{x}^{\star}+\mathbf{z}^{\star}+\mathbf{w}$ with $\boldsymbol{\Theta}=[\mathbf{A},\ \mathbf{I}]\in\mathbb{C}^{m\times(n+m)}$, $\mathbf{A}=\mathbf{U}\mathbf{D}\widetilde{\mathbf{B}}$ and $\mu(\mathbf{U})\sim 1/\sqrt{m}$. If, for $\delta\in(0,1)$,

[TABLE]

where $c_{5}$ and $c_{6}$ are some absolute constants, then with probability at least $1-2\tilde{n}^{-\log^{2}s\log\tilde{n}}$, the $(s,k)$-RIP constant of $\boldsymbol{\Theta}$ satisfies $\delta_{s,k}\leq\delta$.

Proof.

The $(s,k)$-RIP constant $\delta_{s,k}$ can be equivalently expressed as

[TABLE]

where $\mathcal{T}:=\{(\mathbf{x},\mathbf{z}):\|\mathbf{x}\|_{2}^{2}+\|\mathbf{z}\|_{2}^{2}=1,\|\mathbf{x}\|_{0}\leq s,\|\mathbf{z}\|_{0}\leq k,\mathbf{x}\in\mathbb{C}^{n},\mathbf{z}\in\mathbb{C}^{m}\}$. With $\boldsymbol{\Theta}=[\mathbf{A},\ \mathbf{I}]$, the RIP-constant can be further reduced to

[TABLE]

Our aim is to derive bounds on the number of measurements $m$ such that for any $\delta\in(0,1)$ the RIP-constant $\delta_{s,k}$ is upper bounded by $\delta$. We have

[TABLE]

with $\sup_{\mathbf{x}\in\mathcal{D}_{s,n}}\left|\|\mathbf{A}\mathbf{x}\|_{2}^{2}-\|\mathbf{x}\|_{2}^{2}\right|$ being the restricted isometry constant in the standard RIP definition (5). Then, by [7, Theorem III.2], we reach the following result.

Suppose, for any $\delta\in(0,1)$,

[TABLE]

then $\delta_{1}\leq\delta/2$ holds with probability at least $1-\tilde{n}^{-(\log\tilde{n})(\log s)^{2}}$.

Therefore, proof of the $(s,k)$-RIP is reduced to bounding the inner product term $\delta_{2}$.

[TABLE]

where $\mathbf{v}=(\mathbf{z}^{*}\mathbf{U}\text{diag}(\widetilde{\mathbf{B}}\mathbf{x}))^{*}$, and

[TABLE]

The following lemma is proved in Appendix A.

Lemma III.2**.**

Suppose $\boldsymbol{\xi}$ is a length-$\tilde{n}$ random vector with independent, zero-mean, unit-variance, and $L$-subgaussian entries. For any $\delta\in(0,1)$, if

[TABLE]

then $\sup_{\mathbf{v}\in\mathcal{A}_{\mathbf{v}}}|\langle\mathbf{v},\boldsymbol{\xi}\rangle|\leq\delta/2$ holds with probability exceeding $1-\exp(-\log^{2}s\log^{2}\tilde{n})$, where $c_{5}$ and $c_{6}$ are some constants depending only on $L$.

Combining (10) with Lemma III.2, we have, for any $\tau>0$, $\delta_{2}\leq c\delta$ holds with probability exceeding $1-\exp(-\log^{2}s\log^{2}\tilde{n})$ for some constant $c$.

Finally, Theorem III.1 can be obtained by combining the above results. Suppose, for any $\delta\in(0,1)$, $m\geq c_{5}\delta^{-2}s\tilde{n}\mu^{2}(\widetilde{\mathbf{B}})(\log^{2}s\log^{2}\tilde{n})$, $m\geq c_{6}\delta^{-2}k\log^{2}m\log^{2}\tilde{n}$ and $\mu(U)\sim 1/\sqrt{m}$, then we have $\delta_{s,k}\leq\delta_{1}+\delta_{2}\leq\delta$ with probability exceeding

[TABLE]

∎

The uniform recovery guarantee can be obtained by combining Theorem II.2 and III.1.

A few remarks are in order. First, when $\widetilde{\mathbf{B}}$ is a bounded column-wise orthonormal matrix, i.e., $\mu(\widetilde{\mathbf{B}})\sim 1/\sqrt{\tilde{n}}$, the bound on the sparsity of $\mathbf{x}^{\star}$ can be relaxed to $\|\mathbf{x}^{\star}\|_{0}\leq Cm/(\log^{2}\tilde{n}\log^{2}m)$. The sparsity $\|\mathbf{z}^{\star}\|_{0}$ is always a constant fraction of the total number of measurements $m$ regardless the magnitude of the coherence $\mu(\widetilde{\mathbf{B}})$. When $\|\mathbf{w}\|_{2}=0$, Theorem III.1 implies that a sparse signal can be exactly recovered by tractable $l_{1}$ minimization even if some parts of the measurements are arbitrarily corrupted.

Second, the proposed class of structured sensing matrices is equivalent to the UDB framework [7] but with an additional requirement of $\mu(\mathbf{U})\sim 1/\sqrt{m}$. The UDB framework has been proved to support uniform recovery guarantees for conventional CS problem, while with the extra condition it is now shown to provide uniform recovery guarantees for the CS with sparse corruptions problem. Theorem III.1 holds for many existing and new structured sensing matrices as long as they can be decomposed into $\mathbf{A}=\mathbf{U}\mathbf{D}\widetilde{\mathbf{B}}$.

One application of the UDB framework is to simplify the mask design in double random phase encoding (DRPE) for optical image encryption. Consider an image $\mathbf{f}$ that is sparse in the domain $\boldsymbol{\Psi}$, i.e., $\mathbf{f}=\boldsymbol{\Psi}\mathbf{x}^{\star}$, DRPE is based on random masks placed in the input and Fourier planes of the optical system [36, 37] . Mathematically, the measurements can be written as $\mathbf{y}=\sqrt{\frac{n}{m}}\mathbf{R}_{\Omega}\mathbf{F}^{*}\boldsymbol{\Lambda}_{1}\mathbf{F}\boldsymbol{\Lambda}_{2}\mathbf{f}+\mathbf{w}$, where $\mathbf{R}_{\Omega}:\mathbb{C}^{n}\rightarrow\mathbb{C}^{m}$ represents an arbitrary/deterministic subsampling operator with $\Omega$ being the set of selected row indices, $\boldsymbol{\Lambda}_{1}$ and $\boldsymbol{\Lambda}_{2}$ are random diagonal matrices. By the UDB framework, the random diagonal matrix $\boldsymbol{\Lambda}_{2}$ can be replaced by a deterministic diagonal matrix constructed from a Golay sequence $\mathbf{g}$. The reason is that the measurement model $\sqrt{\frac{n}{m}}\mathbf{R}_{\Omega}\mathbf{F}^{*}\boldsymbol{\Lambda}_{1}\mathbf{F}\text{diag}(\mathbf{g})\boldsymbol{\Psi}\mathbf{x}^{\star}$ can be decomposed into a UTF $\sqrt{\frac{n}{m}}\mathbf{R}_{\Omega}\mathbf{F}^{*}$, a random diagonal matrix $\boldsymbol{\Lambda}_{1}$, and a orthonormal matrix $\mathbf{F}\text{diag}(\mathbf{g})\boldsymbol{\Psi}$ whose coherence is proven to be bounded for many orthonormal transforms $\boldsymbol{\Psi}$, e.g., DCT, Haar wavelet [7, Lemma IV.2]. When the measurements are corrupted by impulse noise due to detector plane impairment, our theorem above provides a recovery guarantee on the image.

Furthermore, the UDB framework emcompasses some popular structured sensing matrices, e.g., partial random circulant matrices [14] and random probing [34]. To elaborate, consider the partial random circulant matrices

[TABLE]

where $\mathbf{C}_{\boldsymbol{\epsilon}}$ denotes the circulant matrix generated from $\boldsymbol{\epsilon}$. Suppose $\boldsymbol{\epsilon}=\mathbf{F}^{*}\boldsymbol{\xi}$, where $\mathbf{F}$ is a normalized DFT matrix and $\boldsymbol{\xi}$ is a length-$n$ random vector with independent, zero-mean, unit-variance, and sub-Gaussian entries. Let $\mathbf{D}=\text{diag}(\boldsymbol{\xi})$, we have $\mathbf{A}=\sqrt{\frac{n}{m}}\mathbf{R}_{\Omega}\mathbf{F}^{*}\mathbf{D}\mathbf{F}$. It can be observed that $\mathbf{U}=\sqrt{\frac{n}{m}}\mathbf{R}_{\Omega}\mathbf{F}^{*}$ is a UTF and $\mathbf{B}=\mathbf{F}$ is a unitary matrix. Hence, Theorem III.1 implies that any sparse signal $\mathbf{x}$ and sparse corruption $\mathbf{z}$ can be faithfully recovered from the measurement model $\mathbf{y}=\frac{1}{\sqrt{m}}\mathbf{R}_{\Omega}\mathbf{C}_{\boldsymbol{\epsilon}}\mathbf{x}^{\star}+\mathbf{z}^{\star}+\mathbf{w}$ by the penalized recovery algorithm. The sparse recovery from partial random circulant measurements can be applied in many common deconvolution tasks, such as radar [38] and coded aperture imaging [39]. In practice, where the measurements can be corrupted by impulsive noise due to bit errors in transmission, faulty memory locations, and buffer overflow [40], our theorem guarantees the recovery of both the signal of interest and the corruption.

In some situations, the proposed framework can still provide reliable recovery guarantee even if the corruption is sparse in some basis. Suppose the corruption is sparse under some fixed and known orthonormal transformation $\mathbf{H}$, i.e. $\mathbf{H}^{*}\mathbf{H}=\mathbf{I}$. We consider the measurement model

[TABLE]

It is clear that this setting can be reduced to

[TABLE]

Notice that $\mathbf{H}^{*}\mathbf{A}=\mathbf{H}^{*}\mathbf{U}\mathbf{D}\widetilde{\mathbf{B}}:=\widehat{\mathbf{U}}\mathbf{D}\widetilde{\mathbf{B}}$, where $\widehat{\mathbf{U}}=\mathbf{H}^{*}\mathbf{U}$ is still a UTF due to the orthogonality of $\mathbf{H}$. Therefore, if $\mu(\widehat{\mathbf{U}})\sim 1/\sqrt{m}$, Theorem III.1 still holds in this measurement model.

III-B Randomly sub-sampled orthonormal system

Next, we consider the corrupted sensing measurement model for randomly sub-sampled orthonormal system. We prove the uniform recovery guarantee for such matrices provided that the corruption is sparse on certain sparsifying domain. Suppose $\boldsymbol{\lambda}\in\mathbb{R}^{n}$ is a random Bernoulli vector with i.i.d. entries such that $\mathbb{P}(\lambda_{i}=1)=\frac{m}{n}$ $\forall i\in[n]$ and $\Omega^{\prime}=\{i:\lambda_{i}=1\}$ with $|\Omega^{\prime}|=M$, the random sampling operator $\mathbf{R}_{\Omega^{\prime}}\in\mathbb{R}^{M\times n}$ is a collection of the $i$-th row of an $n$-dimensional identity matrix for all $i\in\Omega^{\prime}$. Here, $M$ is random with mean value $m$. The observation model is

[TABLE]

where $\mathbf{A}=\sqrt{\frac{n}{M}}\mathbf{R}_{\Omega^{\prime}}\mathbf{G}$, $\mathbf{G}\in\mathbb{C}^{n\times n}$ is an orthonormal basis and $\mathbf{H}\in\mathbb{C}^{M\times M}$ is a unitary matrix with $\mu(\mathbf{H})\sim 1/\sqrt{M}$.

From our analysis in previous subsection, the uniform recovery performance can be guaranteed as long as the associated matrix $\boldsymbol{\Theta}$ satisfies the $(s,k)$-RIP. Since the matrix $\mathbf{A}$ satisfies the standard RIP, the problem of proving the $(s,k)$-RIP is again reduced to bounding the inner product term $\sup_{(\mathbf{x},\mathbf{z})\in\mathcal{T}}|\langle\mathbf{A}\mathbf{x},\mathbf{H}\mathbf{z}\rangle|$. Detail proof of the following result is given in Appendix B.

Theorem III.3**.**

Suppose $\mathbf{y}=\mathbf{A}\mathbf{x}^{\star}+\mathbf{H}\mathbf{z}^{\star}+\mathbf{w}$ with $\boldsymbol{\Theta}=[\mathbf{A},\ \mathbf{H}]\in\mathbb{C}^{M\times(n+M)}$, $\mathbf{A}=\sqrt{\frac{n}{M}}\mathbf{R}_{\Omega^{\prime}}\mathbf{G}$ and $\mu(\mathbf{H})\sim 1/\sqrt{M}$. If, for $\delta\in(0,1)$,

[TABLE]

where $\{c_{i}\}_{i=7,...,11}$ are constants, then with probability at least $1-2\tilde{n}^{-\log^{2}s\log n}-n^{-c_{9}}$, the $(s,k)$-RIP constant of $\boldsymbol{\Theta}$ satisfies $\delta_{s,k}\leq\delta$.

When $\mathbf{G}$ is a bounded orthonormal basis, i.e., $\mu(\mathbf{G})\sim 1/\sqrt{n}$, the bound on the sparsity of $\mathbf{x}^{\star}$ can be relaxed to $m\geq\mathcal{O}(\delta^{-2}s\log^{2}s\log^{2}n,\delta^{-2}k\log^{2}k\log^{2}n)$, which implies that a sparse signal can be exactly recovered by tractable $l_{1}$ minimization even if the measurements are affected by corruption sparse on some bounded domain. A bounded orthonormal basis can include the Fourier transform or the Hadamard transform. In addition, in CS-based OFDM where the pilot is generated from a Golay sequence and a random subsampler is employed at the receiver (Section I-A), the effective orthonormal basis is also bounded, i.e., $\mu(\mathbf{F}^{*}\text{diag}(\mathbf{g})\mathbf{F})\sim 1/\sqrt{n}$ [28].

III-C Nonconvex optimization

We have shown the $(s,k)$-RIP for two popular classes of structured sensing matrices, and proven the performance guarantee for the recovery of the sparse signal and corruption via the $l_{1}$-norm minimization algorithm (3). However, our $(s,k)$-RIP analysis on the structured sensing matrices is also applicable to proving the recovery guarantee for nonconvex optimization. Consider the following formulation of the problem

[TABLE]

It was demonstrated in [41] that the unique minimizer of the $l_{p}$ minimization problem ($0<p<1$)

[TABLE]

is exactly the pair $(\mathbf{x}^{\star},\mathbf{z}^{\star})$ if the combined matrix $[\mathbf{A},\mathbf{H}]$ satisfies the $(s,k)$-RIP, where $\nu$ is the regularization parameter. In addition, the $l_{p}$ minimization approach still provides stable recovery even when there is additional dense noise as long as the $(s,k)$-RIP holds [42, 41]. The $l_{p}$ minimization problem can be numerically solved via an iteratively reweighted least squares (IRLS) method [43]. However, [41] only considers the sensing model with $\mathbf{A}$ being random Gaussian matrices and $\mathbf{H}$ being an identity matrix. With our $(s,k)$-RIP analysis, many structured sensing matrices can be employed to provide exact/stable recovery performance for this corrupted sensing problem via $l_{p}$ minimization.

III-D Comparison with related literature

In this part, we compare our main results with related literature.

III-D1 Sparse signal, sparse corruption

[10] proved that sensing matrices with independent Gaussian entries provide uniform recovery guarantee for corrupted CS by solving (3) for all vectors $\mathbf{x}^{\star}$ and $\mathbf{z}^{\star}$ satisfying $\|\mathbf{x}\|_{0}\leq\alpha m/(\log(n/m)+1)$ and $\|\mathbf{z}^{\star}\|_{0}\leq\alpha m$. The difference is that our theorems come with a tighter requirement on the sparsity of $\mathbf{x}^{\star}$ and the sparsity of $\mathbf{z}^{\star}$, which is a compensation on the employment of structured measurements.

[10] also proved the recovery guarantee for structured sensing matrices that belong to the framework proposed in [12]. Here, faithful recovery is guaranteed provided that $\|\mathbf{x}\|_{0}\leq\alpha m/(\mu^{2}\log^{2}n)$ and $\|\mathbf{z}^{\star}\|_{0}\leq\beta m/\mu^{2}$, where $\mu$ is the coherence of the sensing matrix. [11] considered the corrupted CS with sensing matrices that are randomly subsampled orthonormal matrix, and proved similar results. It is noted that the requirements on the sparsity of $\mathbf{x}^{\star}$ in these works seem less strict than that in our results. However, in both [10] and [11], performance guarantees of their structured sensing matrices rely on the assumption that the support set of $\mathbf{x}^{\star}$ or $\mathbf{z}^{\star}$ is fixed and the signs of the signal are independently and equally likely to be $1$ or $-1$ [10, Section 1.3.2][11, Section II.B] (i.e. a nonuniform recovery guarantee). While in our theorem, two classes of structured sensing matrices (including randomly subsampled orthgonal matrix) are shown to provide uniform recovery guarantee for corrupted CS.

We note that recently the uniform recovery guarantee for bounded orthonormal systems is proven in [13]. The bounded orthonormal systems is more general than the random subsampled orthonormal matrix considered in our second class. However, the corruption models are different: the corruption vector in [13] is sparse in time domain, whereas our theorem considers corruption in sparsifying domain with $\mu(\mathbf{H})\sim 1/\sqrt{M}$. Due to this difference in the corruption model, the techniques used to prove the $(s,k)$-RIP (specifically, bound the inner product $\sup_{(\mathbf{x},\mathbf{z})\in\mathcal{T}}|\langle\mathbf{A}\mathbf{x},\mathbf{H}\mathbf{z}\rangle|$) are essentially different.

III-D2 Structured signal, structured corruption

In a recent work [44], sensing with random Gaussian measurements for general structured signals and corruptions (including sparse vectors, low rank matrix, sign vectors and etc) has been proven. However, our study departs from it in the following aspects: [44] proved a nonuniform recovery guarantee for the recovering of sparse signals from sparse corruptions and dense noise. In our paper, we established a uniform recovery guarantee for the corresponding problem. Moreover, [44] considered random Gaussian matrices, while we propose structured sensing matrices.

We have shown that a large class of structured sensing matrices can provide faithful recovery for the sparse sensing with sparse corruption. Whether such structured measurements can be applied in a general corrupted sensing problem (e.g. structured signal with structured corruption) is still open. Extension of our measurement framework to the general corrupted sensing problem is interesting for further study.

Other works related to the recovery of signals from corrupted measurements include [45, 46, 47, 48, 20, 49, 50, 51]. However, their models are different from the one in our paper.

Remark III.4**.**

We note that the value of the regularization parameter can be chosen as $\lambda=\sqrt{s/k}$. In practice, when no a priori knowledge on the sparsity levels of the signal and the corruption is available, $\lambda$ can usually be taken by cross validation. On the other hand, if it is known a priori that the corruption (the signal) is very sparse, one can increase (decrease) the value of $\lambda$ to improve the overall recovery performance. Similar discussion on the theoretical and practical settings of the regularization parameter has also been noted in [10, Section 1.3.3], [11, Section II.E, Section VII], [44, Section III.B]. In addition, an iteratively reweighted $l_{1}$ minimization method can be used to adaptively improve the setting of $\lambda$ in practice [13].**

IV Numerical Simulations

In this section, we verify and reinforce the theoretical results of Section III with a series of simulations. We present experiments to test the recovery performance of the penalized recovery algorithm for the proposed structured sensing matrices. In each experiment, we used the CVX Matlab package [52, 53] to specify and solve the convex recovery programs.

Two different ways of generating sparse vectors are considered:

• •

Gaussian setting: the nonzero entries are drawn from a Gaussian distribution and their locations are chosen uniformly at random,

• •

Flat setting: the magnitudes of nonzero entries are $1$ and their locations are chosen uniformly at random.

IV-A Penalized Recovery

This experiment is to investigate the empirical recovery performance of the penalized recovery algorithm (3) when the dense noise is zero. Here, the sensing matrix (Mtx-I) $\mathbf{A}=\mathbf{U}\mathbf{D}\mathbf{B}$ of size $m\times n$ with $m=256$ and $n=512$ is constructed as below.

Arbitrarily select $m=256$ rows from a $512\times 512$ Hadamard matrix to form a new matrix, which is then normalized by $1/\sqrt{m}$ to form the UTF $\mathbf{U}$. 2. 2.

The diagonal entries of the diagonal matrix $\mathbf{D}$ are i.i.d. Bernoulli random variables. 3. 3.

$\mathbf{B}$ is a normalized Hadamard matrix.

We vary the signal sparsity and the corruption sparsity with $s\in[1,100]$ and $k\in\{10,20,30\}$. For each pair of $(s,k)$, we draw a sensing matrix as described above and perform the following experiment $100$ times:

Generate $\mathbf{x}^{\star}$ with sparsity $s$ by the Gaussian setting 2. 2.

Generate $\mathbf{z}^{\star}$ with sparsity $k$ by the Gaussian setting 3. 3.

Solve (3) by setting $\lambda=1$ 4. 4.

Declare success if444This criterion indicates that both $\mathbf{x}^{\star}$ and $\mathbf{z}^{\star}$ have been faithfully recovered.

[TABLE]

The fraction of successful recovery averaged over the $100$ iterations is presented in Fig. 1a. To demonstrate the performance for signals and corruptions that do not have i.i.d. signs, the experiment is repeated by generating the sparse vectors $\mathbf{x}^{\star}$ and $\mathbf{z}^{\star}$ based on the Flat setting as shown in Fig. 1b. It can be seen that in both scenarios the performance improves as the sparsity of the corruption decreases.

Next, we demonstrate the performance of the penalized recovery algorithm when the sensing matrix is from a randomly subsampled orthonormal matrix. The sensing matrix (Mtx-II) $\mathbf{A}$ is a collection of randomly selected $M=256$ rows from a $512\times 512$ Hadamard matrix, and normalized by $\sqrt{n/M}$. The corruption is $\mathbf{H}\mathbf{z}$, where $\mathbf{H}$ is an $M\times M$ normalized Hadamard matrix. For each pair of $(s,k)$, we repeat the above steps $100$ times to obtain the probability of success (see Fig. 2). It is noted that the recovery performance of Mtx-I is better than that of Mtx-II. This seems consistent with our theoretical analysis as the random subsampled orthonormal matrix shows more stringent recovery condition than the UDB framework (see Theorem III.1 and III.3). However, since the $(s,k)$-RIP is a sufficient condition for the recovery guarantee, it may not fully reflect the performance gap between the two classes of structured sensing matrices. Further investigation based on a necessary and sufficient condition for the recovery guarantee of the corrupted CS problem is a difficult, but interesting open question.

IV-B Stable recovery

We study the stability of the penalized recovery algorithms when the dense noise is nonzero, i.e., $\varepsilon\neq 0$, and compare the performance of structured sensing matrix (Mtx-I) with random Gaussian sensing matrix. In this experiment, the $256$-by-$512$ sensing matrix (Mtx-I) is constructed as in previous subsection. We fix the signal and corruption sparsity levels at $s=10$ and $k=10$ respectively. The dense noise $\mathbf{w}$ consists of i.i.d. Bernoulli entries with amplitude $\varepsilon$. We vary the noise level with $\varepsilon\in[0,0.1]$, and perform the following experiment $100$ times for each $\varepsilon$:

Generate $\mathbf{x}^{\star}$ with $s=10$ by the Gaussian setting 2. 2.

Generate $\mathbf{z}^{\star}$ with $k=10$ by the Gaussian setting 3. 3.

Solve penalized recovery (p-rec) algorithm (3) by setting $\lambda=1$ 4. 4.

Record the empirical recovery error $\|\hat{\mathbf{x}}-\mathbf{x}^{\star}\|_{2}+\|\hat{\mathbf{z}}-\mathbf{z}^{\star}\|_{2}$

An average recovery error is then obtained for each $\varepsilon$. Fig. 3 depicts the average error with varying noise levels. The results in Theorems II.2 and III.1 imply that the recovery errors are bounded by the noise level $\varepsilon$ up to some constants. Fig. 3 clearly shows this linear relationship. In addition, we repeat the above experiments with an iteratively reweighted least squares approach [43] using $p=0.5$. As shown in Fig. 3, the structured sensing matrix is still able to exhibit stable performance by the nonconvex optimization algorithm.

V Conclusion

We have studied a generalized CS problem where the measurement vector is corrupted by both sparse noise and dense noise. We have proven that structured random matrices encompassed in the UDB framework or the randomly subsampled orthonormal system can satisfy the sufficient condition, i.e., the $(s,k)$-RIP. These structured matrices can therefore be applied to provide faithful recovery of both the sparse signal and the corruption by the penalized optimization algorithm as well as the nonconvex optimization algorithm. Our simulations have clearly illustrated and reinforced our theoretical results.

Appendix A Proof of Lemma III.2

Throughout the proof in this and the following sections, $C$ and $c$ denote an absolute constant whose values may change from occurrence to occurrence.

A metric space is denoted by $(T,d)$, where $T$ is a set and $d$ is the notion of distance (metric) between elements of the set. For a metric space $(T,d)$, the covering number $N(T,d,u)$ is the minimal number of open balls of radius $u$ needed to cover $(T,d)$. A subset $\mkern 1.5mu\overline{\mkern-1.5muT\mkern-1.5mu}\mkern 1.5mu$ of $T$ is called a $u$-net of $T$ if every point $\mathbf{x}\in T$ can be approximated to within $u$ by some point $\bar{\mathbf{x}}\in\mkern 1.5mu\overline{\mkern-1.5muT\mkern-1.5mu}\mkern 1.5mu$, i.e., $d(\mathbf{x},\bar{\mathbf{x}})\leq u$. The minimal cardinality of $\mkern 1.5mu\overline{\mkern-1.5muT\mkern-1.5mu}\mkern 1.5mu$ is equivalent to the covering number $N(T,d,u)$. The $p$-th moment (or the $L_{p}$-norm) of a random variable is denoted by $\|X\|_{L_{p}}=(\mathbb{E}|X|^{p})^{1/p}$.

We aim to upper bound the variable $\Delta:=\sup_{\mathbf{v}\in\mathcal{A}_{\mathbf{v}}}|\langle\mathbf{v},\boldsymbol{\xi}\rangle|$ which is the supremum of a stochastic process with the index set $\mathcal{A}_{\mathbf{v}}$. To complete the proof, we require the following important result due to Krahmer et al.:

Theorem A.1**.**

[14, Theorem 3.5 (a)]** Let $\mathcal{A}$ be a set of matrices, and let $\boldsymbol{\xi}$ be a random vector whose entries $\xi_{j}$ are independent, mean [math], variance $1$, and $L$-subgaussian random variables. Set

[TABLE]

Then, for every $p\geq 1$,

[TABLE]

where $C$ is a constant depends only on $L$.

Here, $N_{\mathcal{A}}(\boldsymbol{\xi})$ represents the supremum of certain stochastic processes indexed by a set of matrices $\mathcal{A}$. The above Proposition implies that $N_{\mathcal{A}}(\boldsymbol{\xi})$ can be bounded by three parameters: the suprema of Frobenius norms $d_{F}(\mathcal{A})$, the suprema of operator norms $d_{2\rightarrow 2}(\mathcal{A})$ and a $\gamma_{2}$-functional $\gamma_{2}(\mathcal{A},\|\cdot\|_{2\rightarrow 2})$, which can be bounded in terms of the covering numbers $N(\mathcal{A},\|\cdot\|_{2\rightarrow 2},u)$ as below.

[TABLE]

where the integral is known as Dudley integral or entropy integral [54].

We can transfer the estimates on the moment (17) to a tail bound by the standard estimate due to Markov’s inequality (see [5, Proposition 7.15]).

Proposition A.2**.**

Following the definitions in Theorem A.1, for $t\geq 1$,

[TABLE]

It can be observed that $\Delta$ can be expressed in the form of $N_{\mathcal{A}}(\boldsymbol{\xi})$, where $\mathbf{S}$ and $\mathcal{A}$ are replaced with $\mathbf{v}$ and $\mathcal{A}_{\mathbf{v}}$, respectively. Now, we only need to estimate the parameters $d_{F}(\mathcal{A}_{\mathbf{v}})$, $d_{2\rightarrow 2}(\mathcal{A}_{\mathbf{v}})$ and $\gamma_{2}(\mathcal{A}_{\mathbf{v}},\|\cdot\|_{2\rightarrow 2})$ before bounding $\Delta$ by using Theorem A.2. Since $\mathcal{A}_{\mathbf{v}}$ is a set of vectors, we have $d_{F}(\mathcal{A}_{\mathbf{v}})=d_{2\rightarrow 2}(\mathcal{A}_{\mathbf{v}})$ and $\gamma_{2}(\mathcal{A}_{\mathbf{v}},\|\cdot\|_{2\rightarrow 2})=\gamma_{2}(\mathcal{A}_{\mathbf{v}},\|\cdot\|_{2})$.

For any vector $\mathbf{x}\in\mathcal{D}_{s,n}$, we denote by $\mathbf{x}^{s}$ the length-$s$ vector that retains only the non-zero elements in $\mathbf{x}$. And correspondingly for any vector $\mathbf{b}\in\mathbb{C}^{n}$, we denote by $\mathbf{b}^{s}$ the length-$s$ vector that retains only the elements that have the same indexes as those of the non-zero elements in $\mathbf{x}$. We have, for any $\mathbf{v}\in\mathcal{A}_{\mathbf{v}}$,

[TABLE]

where the last inequality is due to $\|\mathbf{x}\|_{2}^{2}+\|\mathbf{z}\|_{2}^{2}=1$. Therefore,

[TABLE]

Following the same steps, we can alternatively obtain, for any $\mathbf{v}\in\mathcal{A}_{\mathbf{v}}$,

[TABLE]

This provides another upper bound

[TABLE]

We note that both (19) and (20) are valid bounds, and they are not comparable to each other since the relationship between $s$ and $k$ is unknown. It will be clear later that both bounds are useful for computing the entropy integrals. In particular, (19) and (20) are used for computing $I_{1}$ and $I_{2}$ respectively (as in (23)).

Next, we bound $\gamma_{2}$-functional $\gamma_{2}(\mathcal{A}_{\mathbf{v}},\|\cdot\|_{2})$ by estimating the covering numbers $N(\mathcal{A}_{\mathbf{v}},\|\cdot\|_{2},u)$. The derivation is divided into two steps.

Step 1. Decompose $N(\mathcal{A}_{\mathbf{v}},\|\cdot\|_{2},u)$. Let $\mathcal{D}_{1}=\{\mathbf{x}\in\mathbb{C}^{n}:\|\mathbf{x}\|_{2}^{2}\leq 1,\|\mathbf{x}\|_{0}\leq s\}$ and define the semi-norm $\|\cdot\|_{K_{1}}$ as

[TABLE]

For the metric space $(\mathcal{D}_{1},\|\cdot\|_{K_{1}})$, we take $\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{D}_{1}\mkern-1.5mu}\mkern 1.5mu$ to be a $\frac{u}{2}$-net of $\mathcal{D}_{1}$ with $|\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{D}_{1}\mkern-1.5mu}\mkern 1.5mu|=N(\mathcal{D}_{1},\|\cdot\|_{K_{1}},\frac{u}{2})$. Let $\mathcal{D}_{2}=\{\mathbf{z}\in\mathbb{C}^{m}:\|\mathbf{z}\|_{2}^{2}\leq 1,\|\mathbf{z}\|_{0}\leq k\}$ and define the semi-norm $\|\cdot\|_{K_{2}}$ as

[TABLE]

For the metric space $(\mathcal{D}_{2},\|\cdot\|_{K_{2}})$, we take $\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{D}_{2}\mkern-1.5mu}\mkern 1.5mu$ to be a $\frac{u}{2}$-net of $\mathcal{D}_{2}$ with $|\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{D}_{2}\mkern-1.5mu}\mkern 1.5mu|=N(\mathcal{D}_{2},\|\cdot\|_{K_{2}},\frac{u}{2})$.

Now, let $\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{A}_{\mathbf{v}}\mkern-1.5mu}\mkern 1.5mu=\{(\bar{\mathbf{z}}^{*}\mathbf{U}\text{diag}(\widetilde{\mathbf{B}}\bar{\mathbf{x}}))^{*}:\bar{\mathbf{x}}\in\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{D}_{1}\mkern-1.5mu}\mkern 1.5mu,\bar{\mathbf{z}}\in\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{D}_{2}\mkern-1.5mu}\mkern 1.5mu\}$ and remark that $|\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{A}_{\mathbf{v}}\mkern-1.5mu}\mkern 1.5mu|\leq|\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{D}_{1}\mkern-1.5mu}\mkern 1.5mu||\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{D}_{2}\mkern-1.5mu}\mkern 1.5mu|$. It remains to show that for all $\mathbf{v}\in\mathcal{A}_{\mathbf{v}}$, there exists $\bar{\mathbf{v}}\in\bar{\mathcal{A}_{\mathbf{v}}}$ with $\|\mathbf{v}-\bar{\mathbf{v}}\|_{2}\leq u$.

For any $\mathbf{v}=(\mathbf{z}^{*}\mathbf{U}\text{diag}(\widetilde{\mathbf{B}}\mathbf{x}))^{*}\in\mathcal{A}_{\mathbf{v}}$, there exist $\bar{\mathbf{v}}=(\bar{\mathbf{z}}^{*}\mathbf{U}\text{diag}(\widetilde{\mathbf{B}}\bar{\mathbf{x}}))^{*}\in\bar{\mathcal{A}_{\mathbf{v}}}$ with $\bar{\mathbf{x}}\in\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{D}_{1}\mkern-1.5mu}\mkern 1.5mu$ and $\bar{\mathbf{z}}\in\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{D}_{2}\mkern-1.5mu}\mkern 1.5mu$ obeying $\|\mathbf{x}-\bar{\mathbf{x}}\|_{K_{1}}\leq\frac{u}{2}$ and $\|\mathbf{z}-\bar{\mathbf{z}}\|_{K_{2}}\leq\frac{u}{2}$. This gives

[TABLE]

where $(a)$ is due to the fact that $\|\mathbf{z}\|_{2}\leq 1$ and $\|\bar{\mathbf{x}}\|_{2}\leq 1$.

Hence,

[TABLE]

The $\gamma_{2}$-functional $\gamma_{2}(\mathcal{A}_{\mathbf{v}},\|\cdot\|_{2})$ can now be estimated by

[TABLE]

Step 2. Estimate the covering numbers $N(\mathcal{D}_{1},\|\cdot\|_{K_{1}},u/2)$ and $N(\mathcal{D}_{2},\|\cdot\|_{K_{2}},u/2)$ and the entropy integrals. We estimate each covering number in two different ways. For small value of $u$, we use a volumetric argument. For large value of $u$, we use the Maurey method ([14, Lemma 4.2], or [5, Problem 12.9]). Then, the resultant covering number estimates can be used to compute the entropy integrals $I_{1}$ and $I_{2}$. Similar techniques on the covering number estimation and the entropy integral computation have been used in the CS literature, i.e., [6, 14, 55, 7].

From [7, Equation (28)] and (19), we have

[TABLE]

It remains to estimate $N(\mathcal{D}_{2},\|\cdot\|_{K_{2}},u/2)$ and compute $I_{2}$. $1)$ small $u$. We observe that $\mathcal{D}_{2}$ is a subset of the union of $\binom{m}{k}$ unit Euclidean balls $\mathcal{B}_{2}^{k}$,

[TABLE]

For any $\mathbf{z}\in\mathcal{D}_{2}$,

[TABLE]

where the last step is due to the assumption that $\mu(\mathbf{U})\sim\frac{1}{\sqrt{m}}$. Therefore,

[TABLE]

where the last inequality is an application of [6, Proposition 10.1] and [5, Lemma C.5].

$2)$ large $u$. For any $\mathbf{z}\in\mathcal{D}_{2}$, we have $\|\mathbf{z}\|_{1}\leq\sqrt{k}\|\mathbf{z}\|_{2}\leq\sqrt{k}$, which gives

[TABLE]

Then,

[TABLE]

Based on the Maurey method, for $0<u<\frac{1}{2}\mu(\mathbf{U})\sqrt{k}$, the covering number can be estimated by [6, Lemma 8.3]

[TABLE]

We note that the estimation based on Maurey method depends on the range of the parameter $u$ (see [6, Lemma 8.3]), which is the reason why we employ different bounds ((19) and (20)) when computing the entropy integrals $I_{1}$ and $I_{2}$.

We now combine the results (27) and (28) to estimate the entropy integral $I_{2}$: we apply the first bound for $0<u\leq\frac{1}{10}\sqrt{\frac{1}{m}}$, and the second bound for $\frac{1}{10}\sqrt{\frac{1}{m}}<u\leq d_{2\rightarrow 2}(\mathcal{A}_{\mathbf{v}})=\frac{1}{2}\sqrt{\frac{k}{m}}$. It reveals that

[TABLE]

Combine (23), (24) and (29)

[TABLE]

Finally, we are ready to complete the proof by applying Proposition A.2. For the assumption on $m$ and $p$, $\delta\in(0,1)$,

[TABLE]

we have, by (19),

[TABLE]

By substituting the above results into Proposition A.2 (let $t=\log s\log\tilde{n}$), one obtains

[TABLE]

The proof is completed by incorporating the constant $c$ into $c_{1}$, $c_{2}$.

Appendix B Proof of Theorem III.3

Recall that in the measurement model $\mathbf{y}=\mathbf{A}\mathbf{x}^{\star}+\mathbf{H}\mathbf{z}^{\star}+\mathbf{w}$, $\mathbf{A}=\sqrt{\frac{n}{M}}\mathbf{R}_{\Omega^{\prime}}\mathbf{G}$ is a randomly sub-sampled unitary matrix and $\mathbf{H}\in\mathbb{C}^{M\times M}$ is a unitary matrix with $\mu(\mathbf{H})\sim 1/\sqrt{M}$.

The following Lemma from [56] is needed.

Lemma B.1** (Theorem 3.3 [56]).**

For the matrix $\mathbf{A}=\sqrt{\frac{n}{m}}\mathbf{R}_{\Omega^{\prime}}\mathbf{G}$, if for $\delta\in(0,1)$,

[TABLE]

then with probability at least $1-n^{-\log^{3}n}$ the restricted isometry constant $\delta_{s}$ of $\mathbf{A}$ satisfies $\delta_{s}\leq\delta$.

The $(s,k)$-RIP associated with $\boldsymbol{\Theta}=[\mathbf{A},\ \mathbf{H}]$ can be bounded by

[TABLE]

where $\mathcal{T}:=\{(\mathbf{x},\mathbf{z}):\|\mathbf{x}\|_{2}^{2}+\|\mathbf{z}\|_{2}^{2}=1,\|\mathbf{x}\|_{0}\leq s,\|\mathbf{z}\|_{0}\leq k,\mathbf{x}\in\mathbb{C}^{n},\mathbf{z}\in\mathbb{C}^{m}\}$.

By Lemma B.1, we have $\delta_{1}\leq\delta/2$ holds with probability $1-n^{-\log^{3}n}$ for any $\delta\in(0,1)$ if $m\geq c\delta^{-2}s\log^{4}n$.

Define a random vector $\mathbf{d}\in\mathbb{C}^{n}$ with i.i.d. entries satisfying $\lambda_{i}=\sqrt{\frac{m(n-m)}{n^{2}}}d_{i}+\frac{m}{n}$. Assume $\boldsymbol{\Lambda}=\text{diag}(\boldsymbol{\lambda})$ and $\mathbf{H}^{*}\mathbf{R}_{\Omega^{\prime}}=\mathbf{U}^{\prime}$. We have,

[TABLE]

where the last inequality is due to the fact that $\sqrt{\frac{m(n-m)}{n^{2}}}\leq\frac{1}{2}$ for any $m\leq n$.

Since $\boldsymbol{\lambda}$ is a random Bernoulli vector with i.i.d. entries, by construction $\mathbf{d}$ is a length-$n$ random vector with independent, zero-mean, unit-variance, and $L$-subgaussian entries. Hence, the bound for $t_{1}$ can be formulated as the supremum of a stochastic process with the index $\mathcal{A}_{\mathbf{r}}$, where $\mathbf{r}=\sqrt{\frac{n}{M}}\mathbf{z}^{*}\mathbf{U}^{\prime}\text{diag}(\mathbf{G}\mathbf{x})$ and $\mathcal{A}_{\mathbf{r}}:=\{\mathbf{r}:\|\mathbf{x}\|_{2}^{2}+\|\mathbf{z}\|_{2}^{2}=1,\|\mathbf{x}\|_{0}\leq s,\|\mathbf{z}\|_{0}\leq k\}$. For any $\mathbf{r}\in\mathcal{A}_{\mathbf{r}}$,

[TABLE]

Therefore,

[TABLE]

By following the same proof steps as in Appendix A, we have

[TABLE]

provided that

[TABLE]

Bernstein’s inequality [57, Theorem A.1.13] gives, for any $\nu>0$,

[TABLE]

Hence, if

[TABLE]

then

[TABLE]

By assuming $m\geq 2c^{\prime}\log n$ and $\nu=\sqrt{\frac{2c^{\prime}\log n}{m}}$, the above probability of success can be written as

[TABLE]

For the second term, we have

[TABLE]

where the last inequality is due to $\|\mathbf{x}\|_{2}^{2}+\|\mathbf{z}\|_{2}^{2}=1$. Therefore, $t_{2}\leq\delta/2$ for any $\delta\in(0,1)$ if $M\sim m$ and $m\leq\delta n$. By Bernstein’s inequality, this condition can be satisfied with probability exceeding $1-n^{-c^{\prime}}$ as long as $m\leq c\delta^{2}n$. Theorem III.3 is proved by combining the above results.