Greedy Algorithms for Hybrid Compressed Sensing

TL;DR

This paper introduces greedy algorithms for hybrid compressed sensing that combine traditional and one-bit CS to efficiently recover sparse signals under limited bit budgets, especially in noisy conditions.

Contribution

It proposes novel greedy algorithms for hybrid CS, integrating support detection from one-bit CS with residue updates from traditional CS, supported by theoretical analysis and simulations.

Findings

Algorithms perform well under limited bit budgets.

Theoretical proofs support the algorithms' effectiveness.

Simulations confirm robustness in noisy environments.

Abstract

Compressed sensing (CS) is a technique which uses fewer measurements than dictated by the Nyquist sampling theorem. The traditional CS with linear measurements achieves efficient recovery performances, but it suffers from the large bit consumption due to the huge storage occupied by those measurements. Then, the one-bit CS with binary measurements is proposed and saves the bit budget, but it is infeasible when the energy information of signals is not available as a prior knowledge. Subsequently, the hybrid CS which combines the traditional CS and one-bit CS appears, striking a balance between the pros and cons of both types of CS. Considering the fact that the one-bit CS is optimal for the direction estimation of signals under noise with a fixed bit budget and that the traditional CS is able to provide residue information and estimated signals, we focus on the design of greedy algorithms, which consist of the main steps of support detection and recovered signal update, for the hybrid CS in this paper. We first propose a theorem on the random uniform tessellations for sparse signals to further investigate the properties of one-bit CS. Afterwards, we propose two greedy algorithms for the hybrid CS, with the one-bit CS responsible for support detection and traditional CS offering updated residues and signal estimates. For each of the proposed algorithms, we provide the corresponding theorem with proof to analyze their capabilities theoretically. Simulation results have demonstrated the efficacy of the proposed greedy algorithms under a limited bit budget in noisy environments.

Equations

$$[\mathbf{y}{]}_i=⟨{\mathbf{a}}_i,\mathbf{x}⟩,i=1,2,...,m,$$

$$\Vert \mathbf{x}{\Vert }_0:=|\text{mbox}supp(\mathbf{x})|=s\ll n,$$

$$\Vert \mathbf{x}{\Vert }_1\le \sqrt{s}\Vert \mathbf{x}{\Vert }_2,s\ll n.$$

$$[\mathbf{y}{]}_i=\text{mbox}sign(⟨{\mathbf{a}}_i,\mathbf{x}⟩),i=1,2,...,m,$$

$$[\mathbf{y}{]}_i\cdot ⟨{\mathbf{a}}_i,\mathbf{x}⟩\ge 0,i=1,2,...,m,$$

$${\mathbf{y}}_r={\mathbf{A}}_r\tilde{\mathbf{x}}={\mathbf{A}}_r\mathbf{x}+{\mathbf{e}}_r\in {\mathbb{R}}^{m_r},$$

$${\mathbf{y}}_o=\text{mbox}sign({\mathbf{A}}_o\tilde{\mathbf{x}})=\text{mbox}sign({\mathbf{A}}_o\mathbf{x}+{\mathbf{e}}_o)\in \{\pm 1{\}}^{m_o},$$

$$\tilde{\mathbf{x}}=\mathbf{x}+\mathbf{u}\in {\mathbb{R}}^n$$

$$(1-{\delta }_s)\Vert \mathbf{x}{\Vert }_2^2\le \Vert \mathbf{A}\mathbf{x}{\Vert }_2^2\le (1+{\delta }_s)\Vert \mathbf{x}{\Vert }_2^2.$$

$$(1-{\delta }_{|\mathcal{I}|})\Vert \mathbf{x}{\Vert }_2$$

$$\frac{1}{1+{\delta }_{|\mathcal{I}|}}\Vert \mathbf{x}{\Vert }_2$$

$$\Vert {\mathbf{A}}_{{\mathcal{I}}_1}^{\ast }{\mathbf{A}}_{{\mathcal{I}}_2}\mathbf{x}{\Vert }_2\le {\delta }_{|{\mathcal{I}}_1|+|{\mathcal{I}}_2|}\Vert \mathbf{x}{\Vert }_2.$$

$$\mathbf{e}=\mathbf{A}\mathbf{u}\text{mbox}with\Vert \mathbf{u}{\Vert }_1\le d\sqrt{m/\text{mbox}log(n/m)}\Vert \mathbf{e}{\Vert }_2.$$

$${\bf e}={\bf A}{\bf u}\quad\mbox{with}\left\{\begin{array}[]{l}\|{\bf u}\|_{2}\leq d\|{\bf e}\|_{2}\\ \|{\bf u}\|_{1}\leq d^{\prime}\sqrt{m/\mbox{log}(n/m)}\|{\bf e}\|_{2}.\end{array}\right.$$

$$w(\mathcal{K}):=\mathbb{E}\underset{\mathbf{q}\in \mathcal{K}}{\text{mbox}sup}⟨\mathbf{g},\mathbf{q}⟩,$$

$$w^2({\Sigma }_{n,s}^{\prime })\le Cs\text{mbox}log(n/s).$$

$$\text{mbox}conv({\Sigma }_{n,s}^{\prime })\subset {\mathcal{K}}_{n,s}^{\prime }\subset 2\text{mbox}conv({\Sigma }_{n,s}^{\prime }),$$

$$w^2({\mathcal{K}}_{n,s}^{\prime })\le C^{\prime }s\text{mbox}log(n/s).$$

$$w^2(\sqrt{s}{\mathcal{B}}_1^n\cap {\mathcal{S}}^{n-1})\le w^2({\mathcal{K}}_{n,s}^{\prime })\le C^{\prime }s\text{mbox}log(n/s).$$

$$\mathbb{P}\{\text{mbox}sign(⟨\mathbf{a},\mathbf{x}⟩)\ne \text{mbox}sign(⟨\mathbf{a},\mathbf{y}⟩)\}=\frac{\theta (\mathbf{x},\mathbf{y})}{\pi }.$$

$$d_{\mathbf{A}}(\mathbf{x},\mathbf{y}):=|\{i|\text{mbox}sign({\mathbf{a}}_i,\mathbf{x})\ne \text{mbox}sign({\mathbf{a}}_i,\mathbf{y}),i=1,2,...,m\}|/m,$$

$$m\ge C{\delta }^{-4}{w}^2(\mathcal{K}).$$

$$\Vert \mathbf{x}-\mathbf{y}{\Vert }_2\le \delta +d_{\mathbf{A}}(\mathbf{x},\mathbf{y}).$$

$$[{\mathbf{y}}_o{]}_i\cdot ⟨{\mathbf{a}}_{o,i},\tilde{\mathbf{x}}⟩\ge 0,i=1,2,...,m_o.$$

$$m_o\ge C^{\prime }{\delta }^{-4}s\text{mbox}log(n/s).$$

$$\Vert \frac{\mathbf{x}}{\Vert \mathbf{x}{\Vert }_2}-\frac{\hat{\mathbf{x}}}{\Vert \hat{\mathbf{x}}{\Vert }_2}{\Vert }_2\le \delta +d_{{\mathbf{A}}_o}(\tilde{\mathbf{x}},\hat{\mathbf{x}}).$$

$$\prod _{j=2}^s$$

$$\times \prod _{j=1}^s$$

$$-({\kappa }_j-1)[1-I_{{\theta }_{j,2}}(m_o-n_j+1,n_j)]\}$$

$$\prod _{j=1}^s$$

$$[1-I_{{\hat{\theta }}_{j,2}}(m_o-{\hat{n}}_j+1,{\hat{n}}_j)]\}\times \{1-(s-s^{\prime }+j)\times$$

$$I_{{\hat{\theta }}_{j,1}}(m_o-{\tilde{n}}_j+1,{\tilde{n}}_j)-(s^{\prime }-j+1)\times$$

$$[1-I_{{\tilde{\theta }}_{j,2}}(m_o-{\tilde{n}}_j+1,{\tilde{n}}_j)]\}.$$

Full text

I Introduction

With the emergence of the big data world, the amount of information available has dramatically increased during recent years. For practical devices in signal processing and efficient algorithms in machine learning, it is still an open challenge to recover high-dimensional data with a small number of measurements.

Compressed sensing (CS) [1, 2] is a technique which uses a smaller measurement rate than the Nyquist rate to recover the high-dimensional signals which have sparsity. CS is widely adopted in various applications, such as medical imaging [3], video coding [4], radar imaging [5], cognitive radio communications [6], and so on.

In the traditional CS, one would obtain $m$ linear measurements of the form

[TABLE]

where ${\bf a}_{1},{\bf a}_{2},...,{\bf a}_{m}\in{\mathbb{R}}^{n}$ are the $m$ known vectors and ${\bf x}\in{\mathbb{R}}^{n}$ is the signal (which has sparsity) to be recovered. Note that (1) can be rewritten in the form as ${\bf y}={\bf A}{\bf x}$, where the measurement vector ${\bf y}\in{\mathbb{R}}^{m}$ contains the $m$ entries $[{\bf y}]_{1},[{\bf y}]_{2},...,[{\bf y}]_{m}$ and ${\bf A}\in{\mathbb{R}}^{m\times n}$ is the measurement matrix with the rows ${\bf a}_{1},{\bf a}_{2},...,{\bf a}_{m}$.

There are two kinds of sparse signals, $s$-sparse signals and effectively $s$-sparse signals. For a signal ${\bf x}\in{\mathbb{R}}^{n}$, it is called $s$-sparse if

[TABLE]

or it is called effectively $s$-sparse if

[TABLE]

It has been shown that (see the references in [7, 8]) when ${\bf A}$ consists of independent standard normal entries (scaled by any nonzero real constant), one can recover any $s$-sparse signal ${\bf x}$ from the $m\approx s\mbox{log}(n/s)$ linear measurements $[{\bf y}]_{i},i=1,2,...,m$, as defined in (1) with high probability.

Despite its efficient recovery performances, the traditional CS suffers from large bit consumption due to the fact that the linear measurements are stored as numbers of high-precision data types (e.g., float numbers) in most systems. In order to deal with the storage issue, the one-bit CS, which takes binary measurements, is proposed [9].

In the one-bit CS, one would obtain $m$ binary measurements of the form

[TABLE]

where $[{\bf y}]_{1},[{\bf y}]_{2},...,[{\bf y}]_{m}\in\{\pm 1\}$, ${\bf a}_{1},{\bf a}_{2},...,{\bf a}_{m}\in{\mathbb{R}}^{n}$ are the $m$ known vectors, and ${\bf x}\in{\mathbb{R}}^{n}$ is the signal (which has sparsity) to be recovered. Note that (2) can be rewritten in the form as

[TABLE]

and in the form as ${\bf y}=\mbox{sign}({\bf A}{\bf x})$, where the measurement vector ${\bf y}\in\{\pm 1\}^{m}$ contains the $m$ entries $[{\bf y}]_{1},[{\bf y}]_{2},...,[{\bf y}]_{m}$ and ${\bf A}\in{\mathbb{R}}^{m\times n}$ is the measurement matrix with the rows ${\bf a}_{1},{\bf a}_{2},...,{\bf a}_{m}$.

Despite its less bit consumption, the one-bit CS lacks the energy information of signals due to the inherent sign operation. Therefore, the one-bit CS would be infeasible for signal recovery when there is no prior knowledge of signal energy.

Combining both the traditional CS and one-bit CS, the hybrid CS appears in order to strike a balance between the advantages and disadvantages between both types of CS. In our settings of the hybrid CS, one would obtain noisy linear measurements of the form

[TABLE]

and noisy binary measurements of the form

[TABLE]

where

[TABLE]

is the contaminated signal with ${\bf u}\in{\mathbb{R}}^{n}$ being the signal-level noise, ${\bf A}_{r}\in{\mathbb{R}}^{m_{r}\times n}$ and ${\bf A}_{o}\in{\mathbb{R}}^{m_{o}\times n}$ are the measurement matrices of the traditional CS and one-bit CS, respectively, and ${\bf e}_{r}={\bf A}_{r}{\bf u}\in{\mathbb{R}}^{m_{r}}$ and ${\bf e}_{o}={\bf A}_{o}{\bf u}\in{\mathbb{R}}^{m_{o}}$ are the error vectors of the traditional CS and one-bit CS, respectively.

According to the fact that the one-bit CS is optimal for estimating the directions of signals under noise with a fixed bit budget [10] and that the traditional CS can be used to compute residues and signal estimates, it is reasonable to have greedy algorithms, which consist of the main steps of support detection and recovered signal update, for the hybrid CS.

In this study, we first investigate several properties of one-bit CS. Then, we propose two greedy algorithms with proofs for the hybrid CS.

I-A Related Works

Since the hybrid CS is established based on both the traditional CS and one-bit CS, we briefly review the recovery algorithms in both types of CS. In addition, we survey the recent development of the hybrid CS.

For the traditional CS, the recovery algorithms can be classified into three categories: convex relaxation, non-convex optimization, and greedy algorithms [11]. First, the convex relaxation algorithms (e.g., [12, 13, 14, 15, 16, 17]) relax the original problem into convex ones, whose efficient solutions can be derived with existing methods. Next, the non-convex optimization algorithms (e.g., [18, 19]) take advantage of the knowledge of the distribution of the signal to be recovered and provide the statistics of their estimate; however, they are not suitable for the recovery of rather high-dimension signals due to the intractable computational requirements. Lastly, the greedy algorithms (e.g., [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) recover the signal iteratively, choosing a local optimum in each iteration until convergence (or when the stopping criterion is satisfied). A detailed review of these algorithms can be found in several summary papers, e.g., [30, 31].

For the one-bit CS, the recovery algorithms can be classified into three categories: optimization algorithms, Bayesian algorithms, and greedy algorithms. First, the optimization algorithms (e.g., [32, 33, 34, 35, 36, 37, 38, 39, 40, 41]) reformulate the original problem as optimization ones, which can be solved with existing methods efficiently. Next, the Bayesian algorithms (e.g., [42, 43]) adopt the prior knowledge of the signal and provide the estimate with statistical methods. Lastly, the greedy algorithms (e.g., [44, 45]) are executed in an iterative manner, recovering the signal by selecting a local optimum in each iteration until convergence (or when the stop criterion is met).

Integrating both types of CS, the hybrid CS is a rather new topic. To our best knowledge, we only find one work [46] which applies the hybrid CS to the overexposure problem in computed tomography. In [46], the linear measurements whose magnitudes are below the threshold value $s$ will be truncated to the binary values. To solve this hybrid CS problem, the authors propose an optimization algorithm which minimizes the total variation (TV) norm of the estimated signal and the penalty terms. This approach is far different from the greedy algorithms proposed in this study.

I-B Contributions

Focusing on the hybrid CS, this study provides the following three main contributions.

• •

Random uniform tessellations for sparse signals: The one-bit CS is related to the field of random uniform tessellations (see Sec. II-D). Therefore, we propose a fundamental theorem for an analysis of the random uniform tessellations for sparse signals in order to demonstrate the robustness to noise and the properties of error bound of one-bit CS.

• •

Greedy algorithms for hybrid CS: For the hybrid CS, we propose two greedy algorithms, where the one-bit CS is responsible for support detection, and the traditional CS provides the updated residues and signal estimates. The first algorithm detects the support of signals, while the second algorithm modifies an initial guess of the support of signals into a more correct one.

• •

Theoretical analysis of the proposed algorithms: For each of the two proposed algorithms, we offer the corresponding theorem with proof to theoretically analyze its capabilities. For the first algorithm, we provide a theorem on the probability of successful detection of the support of signals; for the second algorithm, we provide a theorem on the probability of successful modifications of the initial guess of the support of signals into the correct one.

I-C Organizations

In Sec. II, we review several essential concepts, such as basic properties of random matrices, Gaussian mean width, and random uniform tessellations. Then, we investigate the properties of random uniform tessellations for sparse signals with the proposed fundamental theorem in Sec. III. In Sec. IV, we introduce the proposed two greedy algorithms for the hybrid CS with corresponding theorems for an analysis of their capabilities. Simulation results and further discussions are provided in Sec. V. Finally, we conclude this study in Sec. VI.

II Preliminaries

In this section, we summary the notations used in this study and review several fundamental concepts related to the proposed algorithms, including basic properties of random matrices, Gaussian mean width, and random uniform tessellations.

II-A Notations

The notations used in this study are summarized as follows: Boldfaced capital and lowercase letters denote matrices and column vectors, respectively. For any set ${\cal A}$, we use $|{\cal A}|$ to denote its cardinality. Given a vector ${\bf v}$, we use $[{\bf v}]_{n}$ to denote the $n$th component of ${\bf v}$, $[{\bf v}]_{{\cal I}}$ the subvector of ${\bf v}$ consisting of the entries indexed by the set ${\cal I}$, $[{\bf v}]^{r}$ the signal formed by restricting ${\bf v}$ to its $r$ largest-magnitude components with the remaining entries set as zero, $\|{\bf v}\|_{2}$ the $\ell_{2}$-norm of ${\bf v}$, $\|{\bf v}\|_{1}$ the $\ell_{1}$-norm of ${\bf v}$, $\|{\bf v}\|_{0}$ the number of nonzero entries of ${\bf v}$. Given two vectors ${\bf v}_{1}$ and ${\bf v}_{2}$, we use $\theta({\bf v}_{1},{\bf v}_{2})$ to denote the angle between ${\bf v}_{1}$ and ${\bf v}_{2}$. Given a matrix ${\bf U}$, we use ${\bf U}_{{\cal I}}$ to denote the submatrix of ${\bf U}$ consisting of the columns indexed by the set ${\cal I}$, ${\bf U}_{\{{\cal I}:0\}}$ the matrix formed by restricting ${\bf U}$ to the columns indexed by the set ${\cal I}^{c}$ with the remaining columns set as zero vectors, ${\bf U}^{*}$ the transpose of ${\bf U}$, ${\bf U}^{\dagger}$ the pseudo inverse of ${\bf U}$. We define ${\cal S}^{n-1}:=\{{\bf v}\in{\mathbb{R}}^{n}\mid\|{\bf v}\|_{2}=1\}$ to be the unit Euclidean sphere in ${\mathbb{R}}^{n}$, and ${\cal B}_{1}^{n}:=\{{\bf v}\in{\mathbb{R}}^{n}\mid\|{\bf v}\|_{1}\leq 1\}$ and ${\cal B}_{2}^{n}:=\{{\bf v}\in{\mathbb{R}}^{n}\mid\|{\bf v}\|_{2}\leq 1\}$ the unit $\ell_{1}$-ball and unit $\ell_{2}$-ball in ${\mathbb{R}}^{n}$, respectively. The set $\Sigma_{n,s}:=\{{\bf v}\in{\mathbb{R}}^{n}\mid\|{\bf v}\|_{0}\leq s\}$ of $s$-sparse vectors in ${\mathbb{R}}^{n}$ is followed by the set $\Sigma^{\prime}_{n,s}=\{{\bf v}\in{\mathbb{R}}^{n}\mid\|{\bf v}\|_{0}\leq s,\|{\bf v}\|_{2}\leq 1\}$ of $s$-sparse vectors in ${\cal B}_{2}^{n}$, while the set ${\cal K}_{n,s}=\{{\bf v}\in{\mathbb{R}}^{n}\mid\|{\bf v}\|_{1}\leq\sqrt{s}\|{\bf v}\|_{2}\}$ of the effectively $s$-sparse vectors in ${\mathbb{R}}^{n}$ is followed by the set ${\cal K}^{\prime}_{n,s}=\{{\bf v}\in{\mathbb{R}}^{n}:\|{\bf v}\|_{1}\leq\sqrt{s},\|{\bf v}\|_{2}\leq 1\}$ of the effectively $s$-sparse vectors in ${\cal B}_{2}^{n}$. We define ${{\bf I}}_{q}$ to be the $q\times q$ identity matrix, and $\tau_{n}({\bf v},{\cal I})$ the vector formed by setting the entries indexed by the set ${\cal I}$ as ${\bf v}$ and the remaining entries as zero. Given a random variable $X$ which follows the binomial distribution $B(n^{\prime},p^{*})$, its cumulative distribution function $\mathbb{P}\{X\leq k\}=I_{1-p^{*}}(n^{\prime}-k,k+1)=(n^{\prime}-k)\binom{n^{\prime}}{k}\int_{0}^{1-p^{*}}t^{n^{\prime}-k-1}(1-t)^{k}dt$ [47].

II-B Basic Properties of Random Matrices

In this study, the measurement matrices ${\bf A}_{r}\in{\mathbb{R}}^{m_{r}\times n}$ and ${\bf A}_{o}\in{\mathbb{R}}^{m_{o}\times n}$ have independent standard normal entries which are divided by $\sqrt{m_{r}}$ and $\sqrt{m_{o}}$, respectively. The following theorems and lemmas introduce several fundamental properties of this kind of random matrix (which will be denoted as ${\bf A}\in{\mathbb{R}}^{m\times n}$ in this subsection). Note that the below $C$ and $c$ are positive constants.

First, we will take a look at the restricted isometry property (RIP), which is one of the most important properties developed in CS.

Definition 1 (RIP of order $s$ [48]): A matrix ${\bf A}$ is said to satisfy the RIP of order $s$ if there exists a constant $\delta_{s}\in(0,1)$ such that the following holds for any ${\bf x}\in\Sigma_{n,s}$:

[TABLE]

*The constant $\delta_{s}$ is called the restricted isometry constant of order $s$. *

*Theorem 1 (Sufficient conditions for RIP of order $s$ [8]): Let $0<\delta_{s}<1$. Suppose that $m\geq C\delta^{-2}s\mbox{log}(n/s)$, then with failure probability at most $2\mbox{exp}(-c\delta^{2}m)$ over the choice of ${\bf A}$, ${\bf A}$ satisfies the RIP of order $s$. *

With the RIP, several properties can be observed through the following lemmas.

*Lemma 1 (Monotonicity of RIP [48]): If a sensing matrix ${\bf A}$ satisfies the RIP of both orders $s_{1}$ and $s_{2}$, then $\delta_{s_{1}}\leq\delta_{s_{2}}$ for any $s1\leq s2$. *

Lemma 2 (Consequences of RIP [48]): Let ${\cal I}\subset\{1,2,...,n\}$. If a sensing matrix ${\bf A}$ satisfies the RIP of order $|{\cal I}|$, then for any ${\bf x}\in{\mathbb{R}}^{|{\cal I}|}$,

[TABLE]

Lemma 3 (Applications of RIP [49]): Let ${\cal I}_{1},{\cal I}_{2}\subset\{1,2,...,n\}$ be two disjoint sets (${\cal I}_{1}\cap{\cal I}_{2}=\varnothing$). If a sensing matrix ${\bf A}$ satisfies the RIP of order $|{\cal I}_{1}|+|{\cal I}_{2}|$, then for any ${\bf x}\in{\mathbb{R}}^{|{\cal I}_{2}|}$,

[TABLE]

Next, we will turn our attention to the $\ell_{1}$-quotient property, which helps with the noise analysis in the CS applications.

Theorem 2 ($\ell_{1}$-quotient property [50]): If $n\geq 2m$, then with failure probability at most $\mbox{exp}(-cm)$, there exists an absolute constant $d$ such that every ${\bf e}\in{\mathbb{R}}^{m}$ can be expressed as

[TABLE]

Combining Theorem 1 and Theorem 2, the following corollary can be derived.

Corollary 1 (Simultaneous $(\ell_{2},\ell_{1})$-quotient property [41]): Suppose that $m\geq C\delta^{-2}s\mbox{log}(n/s)$ and $n\geq 2m$, then with failure probability at most $\mbox{exp}(-cm)$, there exist absolute constants $d$, $d^{\prime}>0$ such that every ${\bf e}\in{\mathbb{R}}^{m}$ can be expressed as

[TABLE]

II-C Gaussian Mean Width

Now, we will have a brief review on the Gaussian mean width. Note that the below $C$ and $C^{\prime}$ are positive constants.

Consider a bounded subset ${\cal K}\subset{\mathbb{R}}^{n}$, the Gaussian mean width of ${\cal K}$ is defined as

[TABLE]

where ${\bf g}\sim N(0,{\bf I}_{n})$ is a standard normal random vector in ${\mathbb{R}}^{n}$.

The Gaussian mean width of several bounded subsets has been studied. For instance, the Gaussian mean width of $\Sigma^{\prime}_{n,s}$ satisfies [36]

[TABLE]

Since ${\cal K}^{\prime}_{n,s}$ is a convexification of $\Sigma^{\prime}_{n,s}$ [38], i.e.,

[TABLE]

the Gaussian mean width of ${\cal K}^{\prime}_{n,s}$ satisfies

[TABLE]

Consider the bounded subset $\sqrt{s}{\cal B}^{n}_{1}\cap{\cal S}^{n-1}\subset{\cal K}^{\prime}_{n,s}$. Its Gaussian mean width satisfies

[TABLE]

II-D Random Uniform Tessellations

Now, we would like to further investigate the one-bit CS which is expressed in the form as (3). Note that $\left\langle{\bf a}_{i},{\bf z}\right\rangle=0,\forall{\bf z}\in{\mathbb{R}}^{n},i=1,2,...,m_{o}$, defines a hyperplane perpendicular to ${\bf a}_{i}$ in ${\mathbb{R}}^{n}$. Suppose that ${\bf x}\in{\cal K}$, where ${\cal K}$ is a subset of ${\mathbb{R}}^{n}$ and can be considered as an oddly shaped object. Assume that there exists another signal ${\bf y}\in{\cal K}$. For a single hyperplane whose normal vector is ${\bf a}_{i},i=1,2,...,m_{o}$, the probability of this hyperplane separating ${\bf x}$ and ${\bf y}$ can be expressed in terms of the angle $\theta({\bf x},{\bf y})$ by the following lemma.

Lemma 4 (Probability of sign change [51]): Let ${\bf x},{\bf y}\in{\mathbb{R}}^{n}$ and ${\bf a}\sim{\cal N}(0,\sigma^{2}{\bf I}_{n})$. Then,

[TABLE]

Before further discussions, we will first introduce the following definition.

Definition 2 (Fraction of hyperplanes that separate points [52]): For ${\bf x},{\bf y}\in{\mathbb{R}}^{n}$, the fraction of $m$ hyperplanes generated from ${\bf A}\in{\mathbb{R}}^{m\times n}$ that separate points ${\bf x}$ and ${\bf y}$ is denoted as

[TABLE]

where ${\bf a}_{i}$ is the $i$th row of ${\bf A}$.

According to Definition 2, if $d_{{\bf A}}({\bf x},{\bf y})$ is smaller, then it can be interpreted as that the object (i.e., ${\cal K}$) is sliced by more hyperplanes, which narrow down the minimum space that contains both ${\bf x}$ and ${\bf y}$ and the bound of the $\ell_{2}$-norm distance between ${\bf x}$ and ${\bf y}$.

The above discussions are linked to the research of random uniform tessellations, which is broadly addressed in [52]. The following theorem provides a connection between $d_{{\bf A}}({\bf x},{\bf y})$ and the upper bound of the $\ell_{2}$-norm distance $\|{\bf x}-{\bf y}\|_{2}$ in the noiseless case. Note that the lower bound of $m$ is relevant to the $w^{2}({\cal K})$.

Theorem 3 (Cells of random uniform tessellations [52]): Consider a bounded subset ${\cal K}\subseteq{\cal S}^{n-1}$ and a matrix ${\bf A}\in{\mathbb{R}}^{m\times n}$, which has independent standard normal entries divided by $\sqrt{m}$. Let $0<\delta<1$. Suppose that

[TABLE]

Then, with probability at least $1-2\mbox{exp}(-c\delta^{4}m)$ over the choice of ${\bf A}$, the following satisfies for any ${\bf x},{\bf y}\in{\cal K}$:

[TABLE]

*The above $C$ and $c$ are positive constants. *

From Theorem 3, it can be observed that the upper bound of the $\ell_{2}$-norm distance $\|{\bf x}-{\bf y}\|_{2}$ is proportional to $d_{{\bf A}}({\bf x},{\bf y})$. Therefore, we can have a smaller $\ell_{2}$-norm distance $\|{\bf x}-{\bf y}\|_{2}$ by decreasing $d_{{\bf A}}({\bf x},{\bf y})$.

III Random Uniform Tessellations for Sparse Signals

According to Sec. II-D, the one-bit CS is closely related to the field of random uniform tessellations. In this section, we would like to further investigate the properties of random uniform tessellations in order to have a better understanding of the one-bit CS. Therefore, we develop a fundamental theorem of random uniform tessellations on the error bound for sparse signals, extended from Sec. II-D, to demonstrate the robustness to noise and the connection between the error bound and the fraction of hyperplanes that separate points of the one-bit CS.

Now, consider the binary measurements, which correspond to the one-bit CS, of the hybrid CS. Note that (5) can be rewritten in the form as

[TABLE]

Assume that the original signal is ${\bf x}\in{\cal K}_{n,s}$. Suppose that there exists an estimate $\hat{{\bf x}}\in{\cal K}_{n,s}$ to ${\bf x}$, then the error bound between ${\bf x}$ and $\hat{{\bf x}}$ can be analyzed, as shown in Theorem 4.

Theorem 4 (Random uniform tessellations for sparse signals): Consider a matrix ${\bf A}_{o}\in{\mathbb{R}}^{m_{o}\times n}$, which has independent standard normal entries divided by $\sqrt{m_{o}}$. Let $0<\delta<1$. Suppose that $n\geq 2m_{o}$, $s\geq Cm_{o}$, and

[TABLE]

Then, with probability at least $1-3\mbox{exp}(-c^{\prime}\delta^{4}m_{o})$ over the choice of ${\bf A}_{o}$, for any ${\bf x}\in{\cal K}_{n,s}$ and ${\bf e}_{o}\in{\mathbb{R}}^{m_{o}}$ with $\|{\bf e}_{o}\|_{2}\leq c\delta^{3}\|{\bf x}\|_{2}$, the estimate $\hat{{\bf x}}\in{\cal K}_{n,s}$ satisfies

[TABLE]

*The above $C$, $C^{\prime}$, $c$, and $c^{\prime}$ are positive constants. *

Proof.

See Appendix A. ∎

From Theorem 4, several properties of the one-bit CS can be observed. Note that the error bound (24) can be lowered by decreasing $d_{{\bf A}_{o}}(\tilde{{\bf x}},\hat{{\bf x}})$, i.e., the estimate $\hat{{\bf x}}$, in replace of $\tilde{{\bf x}}$, satisfies more binary inequalities in (22). In addition, conditioned on the fixed lower bound of the success probability in Theorem 4, the value $\delta$ can be a smaller one if $m_{o}$ increases, resulting in a lower error bound.

IV Proposed Algorithms for Hybrid CS

In the previous section, it is shown that, in the one-bit CS, the error bound between a sparse signal and its sparse estimate can be lowered by decreasing the fraction of hyperplanes generated from the measurement matrix that separate the contaminated signal and the estimate. In addition, the one-bit CS is proved to be optimal for direction estimation of signals under noise with a fixed bit budget [10].

Note that the traditional CS preserves the energy information of signals in its linear measurements. Therefore, the traditional CS can provide residue information and estimated signals.

In this section, we propose two greedy algorithms for the hybrid CS based on the above discussions. Note that in the hybrid CS, one would obtain the noisy linear measurements ${\bf y}_{r}$ and the noisy binary measurements ${\bf y}_{o}$, which are defined in (4) and (5), respectively. The proposed greedy algorithms integrate the advantages of both types of CS, with the one-bit CS responsible for support detection and the traditional CS offering updated residues and signal estimates.

IV-A Binary Inequality Checking with Residue Update

In this subsection, we introduce the first algorithm (see Algorithm 1), which detects the support of the original signal ${\bf x}$ in a greedy manner, and provide a theorem regarding the probability of successful detection.

Algorithm 1 requires the following inputs: the measurement matrix ${\bf A}_{r}$ and the linear measurements ${\bf y}_{r}$ of the traditional CS, the measurement matrix ${\bf A}_{o}$ and the binary measurements ${\bf y}_{o}$ of the one-bit CS, and the sparsity level $s$ of the original signal ${\bf x}$. During the initialization stage, the support is initialized as an empty set, and the residue is set as the linear measurements ${\bf y}_{r}$ of the traditional CS.

In each iteration, Algorithm 1 will pick up a specific index from $\{1,2,...,n\}$ in a greedy manner and add it into the estimated support set. The goal of this algorithm is to detect the support $\Omega$ of the original signal ${\bf x}$ after $s$ iterations.

During the $j$th iteration, $j=1,2,...,s$, Algorithm 1 will execute the following three main steps:

(Step 1) Candidate selection: The detected support set obtained in the $(j-1)$th iteration is denoted as $\Omega_{j-1}$. For each of the indexes in $\Omega_{j-1}^{c}$, compute the magnitude of the inner product between the corresponding column of the matrix ${\bf A}_{r}$ and the residue ${\bf r}_{j-1}$ obtained in the $(j-1)$th iteration individually. Then, sort these values in the decreasing order into a list, and select the column indexes corresponding to the $\left\lfloor\frac{s-j+1}{s}n\right\rfloor$ largest values as candidates ${\cal T}_{j}$ in this iteration. Note that we contract the number of candidates $\left\lfloor\frac{s-j+1}{s}n\right\rfloor$ as the iterations proceed (with $j$ increasing) in accordance with the decreasing number of undetected support positions. For notational brevity, we define $\kappa_{j}:=\left\lfloor\frac{s-j+1}{s}n\right\rfloor$, and we will use $\kappa_{j}$ instead of $\left\lfloor\frac{s-j+1}{s}n\right\rfloor$ hereafter.

(Step 2) Support detection: Now we have the $\kappa_{j}$ candidates in ${\cal T}_{j}$, where we would like to choose an index $\hat{p}$, whose corresponding estimate $\tau_{n}({\bf A}_{r_{\{\Omega_{j-1}\cup\hat{p}\}}}^{\dagger}{\bf y}_{r},\{\Omega_{j-1}\cup\hat{p}\})$, in replace of $\tilde{{\bf x}}$, satisfies the most binary inequalities in (22), as the index detected in this iteration. Then, we add $\hat{p}$ into the detected support set and obtain $\Omega_{j}$.

(Step 3) Residue update: With the detected support set $\Omega_{j}$, we derive the residue ${\bf r}_{j}$ for this iteration.

The following theorem provides an analysis of Algorithm 1 on detecting $\Omega$ after $s$ iterations.

Theorem 5 (Successful detection with Algorithm 1): Let ${\bf A}_{r}\in{\mathbb{R}}^{m_{r}\times n}$ and ${\bf A}_{o}\in{\mathbb{R}}^{m_{o}\times n}$ have independent standard normal entries, which are divided by $\sqrt{m_{r}}$ and $\sqrt{m_{o}}$, respectively. Consider any ${\bf x}\in\Sigma_{n,s}$ whose support $\Omega=\{i_{1},i_{2},...,i_{s}\}$ satisfies $|[{\bf x}]_{i_{1}}|\geq|[{\bf x}]_{i_{2}}|\geq...\geq|[{\bf x}]_{i_{s}}|$. Assume that ${\bf A}_{r}$ satisfies $\delta_{n}\in(0,0.5]$ and $\|{\bf e}_{r}\|_{2}\leq\frac{\|{\bf x}\|_{2}-\sqrt{2}\|[{\bf x}]_{\Omega\setminus i_{1}}\|_{2}}{2+\sqrt{2}}$ with $\frac{\|{\bf x}\|_{2}}{\|[{\bf x}]_{\Omega\setminus i_{1}}\|_{2}}\geq\sqrt{2}$. Let $c_{j}>0$ and $n_{j}\in{\mathbb{N}}$ be the reference value of candidate selection and the threshold number of support detection in the $j$th iteration. Then, Algorithm 1 detects $\Omega$ after $s$ iterations with probability at least

[TABLE]

where $\theta_{j,1}=\frac{\mbox{sin}^{-1}(1-\frac{\sqrt{2}(\|[{\bf x}]_{\Omega\setminus i_{1}}\|_{2}-\|[{\bf x}]_{\Omega\setminus\{i_{1},i_{2},...,i_{j}\}}\|_{2})}{\|{\bf x}\|_{2}})}{\pi}$ and $\theta_{j,2}=\frac{\mbox{cos}^{-1}(\frac{\|[{\bf x}]_{\{i_{1},i_{2},...,i_{j-1},i_{j+1}\}}\|_{2}}{\|{\bf x}\|_{2}})}{\pi}$.

Proof.

See Appendix B. ∎

IV-B Support Modification via Binary Inequality Checking

In this subsection, we will present the second algorithm, as described in Algorithm 2, which refines an initial guess of the support of the original signal ${\bf x}$ in a greedy manner until convergence, and provide a theorem regarding the probability of successful modification.

Algorithm 2 requires the measurement matrix ${\bf A}_{r}$ and the linear measurements ${\bf y}_{r}$ of the traditional CS, the measurement matrix ${\bf A}_{o}$ and the binary measurements ${\bf y}_{o}$ of the one-bit CS, the sparsity level $s$ of the original signal ${\bf x}$, and an initial guess of the support of ${\bf x}$, $\Omega_{0}$, with $|\Omega_{0}|=s$ as inputs. During the initialization stage, we only set the iteration index $j$ as 1, and there is no need to initialize any other parameter.

In each iteration, we will first select a new index and add it into the estimated support set, and then eliminate an index from the estimated support set. As a result, the cardinality of the estimated support set will always be equal to $s$ after arbitrary times of iterations.

During the $j$th iteration, $j\in{\mathbb{N}}$, Algorithm 2 executes the following two main steps:

(Step 1) Support augmentation: With the estimated support set $\Omega_{j-1}$ obtained in the $(j-1)$th iteration, we would like to choose an index $\hat{p}\in\Omega_{j-1}^{c}$, whose corresponding estimate $\tau_{n}({\bf A}_{r_{\{\Omega_{j-1}\cup\hat{p}\}}}^{\dagger}{\bf y}_{r},\{\Omega_{j-1}\cup\hat{p}\})$, in replace of $\tilde{{\bf x}}$, satisfies the most binary inequalities in (22), from the $|\Omega_{j-1}^{c}|=n-s$ candidates, as the index appended to the estimated support set. Consequently, we would obtain the augmented estimated support set ${\cal T}_{j}$.

(Step 2) Support pruning: Now we have the augmented estimated support set ${\cal T}_{j}$ with $|{\cal T}_{j}|=s+1$. Then, we would like to obtain a pruned estimated support set $\Omega_{j}\subset{\cal T}_{j}$ with $|\Omega_{j}|=s$, from the $\binom{s+1}{s}=s+1$ candidates, such that $\tau_{n}({\bf A}_{r_{\Omega_{j}}}^{\dagger}{\bf y}_{r},\Omega_{j})$, in replace of $\tilde{{\bf x}}$, satisfies the most binary inequalities in (22).

Note that the stopping criterion of Algorithm 2 is that $\Omega_{j}=\Omega_{j-1}$, which results in a convergence. In the following, Theorem 6 provides an analysis of Algorithm 2 on refining the initial guess $\Omega_{0}$ of the support of ${\bf x}$ into $\Omega$.

Theorem 6 (Successful Modification with Algorithm 2): Let ${\bf A}_{r}\in{\mathbb{R}}^{m_{r}\times n}$ and ${\bf A}_{o}\in{\mathbb{R}}^{m_{o}\times n}$ have independent standard normal entries, which are divided by $\sqrt{m_{r}}$ and $\sqrt{m_{o}}$, respectively. Consider any ${\bf x}\in\Sigma_{n,s}$ whose support $\Omega=\{i_{1},i_{2},...,i_{s}\}$. Suppose that the initial guess $\Omega_{0}$ fails to detect $s^{\prime}$ support indexes $\{i_{1},i_{2},...,i_{s^{\prime}}\}$ with $|[{\bf x}]_{i_{1}}|\geq|[{\bf x}]_{i_{2}}|\geq...\geq|[{\bf x}]_{i_{s^{\prime}-1}}|\geq|[{\bf x}]_{i_{s^{\prime}}}|$, i.e., the set of initial detected support indexes is $\tilde{\Omega}_{0}=\{i_{s^{\prime}+1},i_{s^{\prime}+2},...,i_{s^{\prime}}\}$. Assume that ${\bf A}_{r}$ satisfies $\delta_{n}\in(0,0.5]$ and $\|{\bf e}_{r}\|_{2}\leq\frac{\|{\bf x}\|_{2}-\sqrt{2}\|[{\bf x}]_{\Omega\setminus\{\tilde{\Omega}_{0}\cup i_{1}\}}\|_{2}}{2+\sqrt{2}}$ with $\frac{\|{\bf x}\|_{2}}{\|[{\bf x}]_{\Omega\setminus\{\tilde{\Omega}_{0}\cup i_{1}\}}\|_{2}}\geq\sqrt{2}$. Let $\hat{n}_{j},\tilde{n}_{j}\in{\mathbb{N}}$ be the threshold number of support augmentation and support pruning, respectively, in the $j$th iteration. Then, Algorithm 2 modifies $\Omega_{0}$ into $\Omega$ after $s^{\prime}$ iterations with probability at least

[TABLE]

where $\hat{\theta}_{j,1}=\frac{\mbox{sin}^{-1}(1-\frac{\sqrt{2}(\|[{\bf x}]_{\Omega\setminus\{\tilde{\Omega}_{0}\cup i_{1}\}}\|_{2}-\|[{\bf x}]_{\Omega\setminus\{\tilde{\Omega}_{0},i_{1},i_{2},...,i_{j-1}\}}\|_{2})}{\|{\bf x}\|_{2}})}{\pi}$, $\hat{\theta}_{j,2}=\frac{\mbox{cos}^{-1}(\frac{\|[{\bf x}]_{\{\tilde{\Omega}_{0},i_{1},i_{2},...,i_{j-1},i_{j+1}\}}\|_{2}}{\|{\bf x}\|_{2}})}{\pi}$, and $\tilde{\theta}_{j,2}=\frac{\mbox{cos}^{-1}(\frac{\|[{\bf x}]_{\{\tilde{\Omega}_{0},i_{1},i_{2},...,i_{j}\}\setminus i_{j}^{*}}\|_{2}}{\|{\bf x}\|_{2}})}{\pi}$ with $i_{j}^{*}=\underset{i\in\{\tilde{\Omega}_{0},i_{1},i_{2},...,i_{j}\}}{\mbox{argmin}}|[{\bf x}]_{i}|$.

Proof.

See Appendix C. ∎

V Simulation

In this section, we present the numerical results in order to compare the recovery performances of the proposed greedy algorithms of hybrid CS with those of the classic greedy algorithms of traditional CS for two experiments with each algorithm sharing the same bit budget. Note that the greedy algorithms of one-bit CS are not considered for the simulations since they require the prior knowledge of the signals’ energy information, which is hardly known in real-world applications. The greedy algorithms of traditional CS that are included in the experiments are the orthogonal matching pursuit (OMP) [20], subspace pursuit (SP) [23], and CoSaMP [24].

V-A Parameter Settings

Each linear measurement in traditional CS is stored as a float number which consumes 32 bits, while each binary measurement in one-bit CS consumes only 1 bit. For each test of each experiment, we adopt the Monte Carlo simulation by randomly generating $n_{v}=500$ $s$-sparse vectors of dimension $n=256$, with each vector containing standard normal entries in its support, as the signal ${\bf x}$ to be recovered. With each signal generated, we construct corresponding measurement matrices ${\bf A}_{r}\in{\mathbb{R}}^{m_{r}\times n}$ and ${\bf A}_{o}\in{\mathbb{R}}^{m_{o}\times n}$ for the proposed greedy algorithms of hybrid CS, and ${\bf A}\in{\mathbb{R}}^{m\times n}$ for the greedy algorithms of traditional CS. Note that these measurement matrices contain the standard normal entries divided by the square root of the number of measurements. Also, Algorithm 2 takes the estimated support set obtained from Algorithm 1 as the initial guess. There are two types of signal-to-noise ratio (SNR) used in the simulations, signal SNR $\xi_{s}:=10\mbox{log}\frac{\|{\bf x}\|_{2}^{2}}{\|{\bf u}\|_{2}^{2}}$ (dB) and recovery SNR $\xi_{r}:=10\mbox{log}\frac{1}{n_{v}}\sum_{i=1}^{n_{v}}\frac{\|{\bf x}_{i}\|_{2}^{2}}{\|{\bf x}_{i}-\hat{{\bf x}}_{i}\|_{2}^{2}}$ (dB), where ${\bf x}_{i}$ and $\hat{{\bf x}}_{i}$ are the $i$th signal and estimate in a single test. Note that the recovery performances are evaluated in terms of $\xi_{r}$.

The first experiment evaluates the recovery performances of all greedy algorithms under a relatively small budget of $64s$ bits in the noisy case with the tests executed across $s=\{4,8,16,32\}$ and $\xi_{s}=\{0,5,10,15,20,25,30\}$. In the first experiment, $m_{r}=\left\lceil 1.5s\right\rceil$ and $m_{o}=32\times\left\lfloor 0.5s\right\rfloor$ for the greedy algorithms of hybrid CS, and $m=2s$ for those of traditional CS.

The second experiment evaluates the recovery performances of all greedy algorithms under a fixed budget of $64\times 32=2048$ bits in the noisy case with the tests executed across $s=\{4,8,16,32\}$ and $\xi_{s}=\{0,10\}$. In the second experiment, $m_{r}=48$ and $m_{o}=16\times 32=512$ for the greedy algorithms of hybrid CS, and $m=64$ for those of traditional CS.

V-B Simulation Results

For the first experiment, the simulation results are shown in Fig. 1, where we compare the recovery performances of the proposed greedy algorithms of hybrid CS with those of the greedy algorithms of traditional CS in the noisy case under a relatively small bit budget. It can be observed that the proposed greedy algorithms outperform the greedy algorithms of traditional CS, which shows the robustness to noise and the feasibility under a small bit budget of the proposed algorithms. This is because the proposed algorithms strike a balance between the pros and cons of both the traditional CS and one-bit CS. Particularly, the traditional CS can be regarded as a special case of the hybrid CS. Note that the recovery performances of Algorithm 2 is better than those of Algorithm 1, which demonstrates the efficacy of Algorithm 2 on modifying the initial guess of the original signal’s support into a more correct one. In addition, it can be found that the recovery performances of all greedy algorithms degrade when $s$ increases or when the signal SNR decreases.

For the second experiment where the bit budget is fixed, the simulation results are shown in Fig. 2 and exhibit a similar trend to the first experiment.

VI Conclusions

In this study, we focus on the hybrid compressed sensing (CS), which combines both the traditional CS and one-bit CS. First, we propose a fundamental theorem on the random uniform tessellations for sparse signals in order to investigate the properties of one-bit CS. Then, we propose two greedy algorithms for the hybrid CS, with the one-bit CS detecting the support indexes and the traditional CS providing the updated residues and signal estimates. For each of the two algorithms, we offer the corresponding theorem to theoretically analyze its capabilities. Numerical results have demonstrated the efficacy of the proposed two algorithms, compared with classic greedy algorithms of traditional CS. The future directions of the hybrid CS include the optimal split of bit budget for linear measurements and binary measurements, and Bayesian algorithms with statistical methods.

Acknowledgment

This work was supported by Ministry of Science and Technology, Taiwan, under Grants MOST 107-2221-E-001-015-MY2 and MOST 108-2634-F-007-010.

Appendix A Proof of Theorem 4

First, note that $\|{\bf e}_{o}\|_{2}\leq c\delta^{3}\|{\bf x}\|_{2}$. According to Corollary 1, with failure probability at most $\mbox{exp}(-cm_{o})$, there exists a vector ${\bf u}$ satisfying

[TABLE]

Note that $n/m_{o}\geq 2$, and thus $\mbox{log}(n/m_{o})\geq\mbox{log}2$. Therefore,

[TABLE]

where the last inequality holds since $s\geq Cm_{o}$.

With the triangular inequality, the upper bound and the lower bound of the $\ell_{2}$-norm of the contaminated signal $\tilde{{\bf x}}$ can be derived as

[TABLE]

[TABLE]

In a similar manner, the upper bound of the $\ell_{1}$-norm of $\tilde{{\bf x}}$ can be derived as

[TABLE]

Combining (30) and (31), the upper bound of the $\ell_{1}$-norm of the normalized contaminated signal $\frac{\tilde{{\bf x}}}{\|\tilde{{\bf x}}\|_{2}}$ can be obtained as

[TABLE]

Therefore, the normalized contaminated signal $\frac{\tilde{{\bf x}}}{\|\tilde{{\bf x}}\|_{2}}\in 2\sqrt{s}{\cal B}^{n}_{1}\cap{\cal S}^{n-1}$. Note that $\frac{\hat{{\bf x}}}{\|\hat{{\bf x}}\|_{2}}\in\sqrt{s}{\cal B}_{1}^{n}\cap{\cal S}^{n-1}$, and therefore $\frac{\hat{{\bf x}}}{\|\hat{{\bf x}}\|_{2}}\in 2\sqrt{s}{\cal B}_{1}^{n}\cap{\cal S}^{n-1}$.

According to (17) and Theorem 3, with probability at least $1-2\mbox{exp}(-c\delta^{4}m_{o})$, the following satisfies:

[TABLE]

given $m_{o}\geq C^{\prime}\delta^{-4}s\mbox{log}(n/s)$.

We target the inequality

[TABLE]

where the first inequality holds due to the inequality (30), and the last inequality holds due to the upper bound of $\|{\bf u}\|_{2}$ obtained in (27). In addition, it can be derived that

[TABLE]

where the first inequality holds due to the upper bound and lower bound of $\|\tilde{{\bf x}}\|_{2}$ as stated in (29) and (30), and the last equality holds due to the fact that $\delta$ is chosen from the interval $(0,1]$.

Finally, with the triangular inequality, we obtain that

[TABLE]

where the first inequality holds due to the triangular inequality, and the last inequality holds due to the inequalities (33), (34), and (35).

Appendix B Proof of Theorem 5

Without loss of generality, assume that the first $s$ entries of ${\bf x}$ are nonzero with $|[{\bf x}]_{1}|\geq|[{\bf x}]_{2}|\geq...\geq|[{\bf x}]_{s}|$, while the remaining $n-s$ entries are equal to zero. Therefore, we denote the support set and the non-support set of ${\bf x}$ as $\Omega=\{1,2,...,s\}$ and ${\cal N}{\cal S}=\{s+1,s+2,...,n\}$, respectively.

Now, we define two events, $E_{\footnotesize{\mbox{succ}}}$: the support set $\Omega$ is detected after the $s$ iterations and $E_{\varnothing\rightarrow a^{\prime}}$: the order of detected indexes after the first $a^{\prime}$ iterations is $1\rightarrow 2\rightarrow...\rightarrow a^{\prime}$. Consider a specific event $E_{\varnothing\rightarrow s}$ where the order of detected indexes after the $s$ iterations is: $1\rightarrow 2\rightarrow...\rightarrow s$. Note that when the event $E_{\varnothing\rightarrow s}$ happens, the event $E_{\footnotesize{\mbox{succ}}}$ also happens. Therefore, it can be inferred that

[TABLE]

With the multiplication law of probability, $\mathbb{P}\{E_{\varnothing\rightarrow s}\}$ can be decomposed as

[TABLE]

where the event $E_{a^{\prime}-1\rightarrow a^{\prime}}$ is defined as: the detected index in the $(a^{\prime}-1)$th iteration is $a^{\prime}-1$ and the detected index in the $a^{\prime}$th iteration is $a^{\prime}$.

Note that the first term on the RHS of (38) corresponds to the detection of the index 1 in the first iteration, while the $a^{\prime}$th term on the RHS of (38) corresponds to the detection of the index $a^{\prime}$ in the $a^{\prime}$th iteration, given that the order of detected indexes in the previous $a^{\prime}-1$ iterations is: $1\rightarrow 2\rightarrow...\rightarrow a^{\prime}-1$, $a^{\prime}=2,3,...,s$.

Define two events, $E_{\footnotesize{\mbox{cand}},a^{\prime}}$: the index $a^{\prime}$ is selected as one of the $\kappa_{a^{\prime}}$ candidates in ${\cal T}_{a^{\prime}}$ in the $a^{\prime}$th iteration and $E_{\footnotesize{\mbox{supp}},a^{\prime}}$: the index $a^{\prime}$ is detected in the $a^{\prime}$th iteration. Then, with the multiplication law of probability, the first term and the $a^{\prime}$th term, $a^{\prime}=2,3,...,s$, on the RHS of (38) can be decomposed as

[TABLE]

[TABLE]

respectively. In the following proof, we will address the probability terms on the RHS of (39) and (40).

First, consider the step of candidate selection. In the $j$th iteration, we would select the $\kappa_{j}$ candidates from $\Omega_{j-1}^{c}$ as ${\cal T}_{j}$, which should contain the index $j$ so that we may detect it in the next step.

When $j=1$, we would select $\kappa_{1}=n$ candidates from $\Omega_{0}^{c}=\{1,2,...,n\}$. It is obvious that the index 1 must be chosen as one of the candidates in ${\cal T}_{1}$. Therefore, we derive that

[TABLE]

When $j=2,3,...,s$, the detected support set obtained in the previous $j-1$ iterations is $\Omega_{j-1}=\{1,2,...,j-1\}$, conditioned on the event $E_{\varnothing\rightarrow j-1}$. Note that we would like to have the index $j$ as one of the candidates in ${\cal T}_{j}$. Therefore, we still need to select $\kappa_{j}-1$ candidates, i.e., eliminate $(n-j)-(\kappa_{j}-1)=n-j+1-\kappa_{j}$ indexes, from $\{\Omega_{j-1}\cup j\}^{c}$.

Now, consider the event $E_{j,c_{j}}$ where $|\left\langle{\bf a}_{r,j},\frac{\sqrt{m_{r}}{\bf r}_{j-1}}{\|{\bf r}_{j-1}\|_{2}}\right\rangle|>c_{j}$ and at least $n-j+1-\kappa_{j}$ indexes from $\{\Omega_{j-1}\cup j\}^{c}$ satisfy $|\left\langle{\bf a}_{r,i},\frac{\sqrt{m_{r}}{\bf r}_{j-1}}{\|{\bf r}_{j-1}\|_{2}}\right\rangle|\leq c_{j},i\in\{\Omega_{j-1}\cup j\}^{c}$, where $c_{j}$ is the reference value for candidate selection in the $j$th iteration and ${\bf a}_{r,i^{\prime}}$ is the $i^{\prime}$th column of the matrix ${\bf A}_{r}$. Conditioned on the event $E_{\varnothing\rightarrow j-1}$, when the event $E_{j,c_{j}}$ happens, the event $E_{\footnotesize{\mbox{cand}},j}$ also happens. Consequently, it can be inferred that

[TABLE]

Note that conditioned on the event $E_{\varnothing\rightarrow j-1}$, $\left\langle{\bf a}_{r,i^{\prime}},\frac{\sqrt{m_{r}}{\bf r}_{j-1}}{\|{\bf r}_{j-1}\|_{2}}\right\rangle\sim N(0,1),i^{\prime}=1,2,...,n$. Therefore, we have

[TABLE]

where the inequalities in (43) and (44) hold due to the bounds for standard normal distribution.

Denote the number of indexes in $\{\Omega_{j-1}\cup j\}^{c}$ that satisfy $|\left\langle{\bf a}_{r,i},\frac{\sqrt{m_{r}}{\bf r}_{j-1}}{\|{\bf r}_{j-1}\|_{2}}\right\rangle|\leq c_{j},i\in\{\Omega_{j-1}\cup j\}^{c}$ conditioned on the event $E_{\varnothing\rightarrow j-1}$ as $B_{\footnotesize{\mbox{Low}},j}\sim B(n-j,p_{\footnotesize{\mbox{Low}},j})$, where $p_{\footnotesize{\mbox{Low}},j}\geq 1-e^{-\frac{{c_{j}}^{2}}{2}}$, which follows (44).

Now, we can compute the probability

[TABLE]

where the first inequality holds due to (43).

Next, consider the step of support detection. In the $j$th iteration, we would like to detect the index $j$ from ${\cal T}_{j}$, conditioned on the event $E_{\footnotesize{\mbox{cand}},j}$.

Now, we define two events, $E_{\Omega_{1},n_{1}}$: when $j=1$, $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{1,1}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|\geq n_{1}$ and $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{1,p^{\prime}}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|\leq n_{1}-1,p^{\prime}\in\{\Omega_{0}\cup 1\}^{c}$, conditioned on the event $E_{\footnotesize{\mbox{cand}},1}$, and $E_{\Omega_{a^{\prime}},n_{a^{\prime}}}$: when $j=a^{\prime},a^{\prime}=2,3,...,s$, $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{a^{\prime},a^{\prime}}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|\geq n_{a^{\prime}}$ and $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{a^{\prime},p^{\prime}}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|\leq n_{a^{\prime}}-1,p^{\prime}\in\{\Omega_{a^{\prime}-1}\cup a^{\prime}\}^{c}$, conditioned on the events $E_{\footnotesize{\mbox{cand}},a^{\prime}}$ and $E_{\varnothing\rightarrow a^{\prime}-1}$, where $n_{j},j=1,2,...,s$ is the threshold number of support detection in the $j$th iteration.

Note that when the event $E_{\Omega_{1},n_{1}}$ happens, the event $E_{\footnotesize{\mbox{supp}},1}$ conditioned on the event $E_{\footnotesize{\mbox{cand}},1}$ also happens, and when the event $E_{\Omega_{a^{\prime}},n_{a^{\prime}}}$ happens, the event $E_{\footnotesize{\mbox{supp}},a^{\prime}}$ conditioned on the events $E_{\footnotesize{\mbox{cand}},a^{\prime}}$ and $E_{\varnothing\rightarrow a^{\prime}-1}$ also happens. Therefore, it can be inferred that

[TABLE]

[TABLE]

where the event $E_{\Omega_{j},n_{j}}$ implies that the detected support set in the $j$th iteration would be $\Omega_{j}=\{1,2,...,j\}=\{\Omega_{j-1},j\}$ ($\Omega_{0}=\varnothing$).

In order to analyze the probability of the hyperplanes generated from ${\bf A}_{o}$ separating ${\bf x}$ and the estimate $\tau_{n}({\bf A}^{\dagger}_{r_{\Omega_{j}}}{\bf y}_{r},\Omega_{j})$, compute the $\ell_{2}$-norm distance between both points as

[TABLE]

where the inequality holds due to the triangular inequality. Note that the first term and the second term on the RHS of (48) correspond to the signal-level noise and the distance between $\tilde{{\bf x}}$ and the estimate.

We can compute the square of the second term on the RHS of (48) as

[TABLE]

where the first inequality holds due to Lemma 2 and the second inequality holds due to Lemma 3.

Note that the matrix ${\bf A}_{r}$ satisfies $\delta_{n}\in(0,0.5]$ and $\delta_{j}\leq\delta_{n}$ (which is implied by Lemma 1). According to Definition 1, since $\|{\bf e}_{r}\|_{2}\leq\frac{\|{\bf x}\|_{2}-\sqrt{2}\|[{\bf x}]_{\Omega\setminus 1}\|_{2}}{2+\sqrt{2}}$, it can be inferred that

[TABLE]

With (49), (50), and the properties of ${\bf A}_{r}$, (48) becomes

[TABLE]

where the second inequality holds due to the triangular inequality. Note that $\|{\bf x}\|_{2}-\sqrt{2}(\|[{\bf x}]_{\Omega\setminus 1}\|_{2}-\|[{\bf x}]_{\Omega\setminus\Omega_{j}}\|_{2})\leq\|{\bf x}\|_{2}$.

Consider the hyperplane with normal vector ${\bf a}_{o,i},i=1,2,...,m_{o}$, which is the $i$th row of the matrix ${\bf A}_{o}$. First, compute the probability of the hyperplane not separating ${\bf x}$ and the estimate obtained from $\Omega_{j}$. According to (51) and Lemma 4, we have

[TABLE]

Next, compute the probability of the hyperplane not separating ${\bf x}$ and the estimate obtained from $\{\Omega_{j-1}\cup p^{\prime}\},p^{\prime}\in{\cal T}_{j}-j$. According to Lemma 4, we have

[TABLE]

Denote $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{j,j}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|$ and $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{j,p^{\prime}}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|,p^{\prime}\in{\cal T}_{j}-j$, conditioned on the same events as $E_{\Omega_{j},n_{j}}$ is, as $B_{j}\sim B(m_{o},p_{j}),p_{j}\geq 1-\frac{\mbox{sin}^{-1}(1-\frac{\sqrt{2}(\|[{\bf x}]_{\Omega\setminus 1}\|_{2}-\|[{\bf x}]_{\Omega\setminus\Omega_{j}}\|_{2})}{\|{\bf x}\|_{2}})}{\pi}$ and $B_{j,p^{\prime}}\sim B(m_{o},p_{j,x}),p_{j,x}\leq 1-\frac{\mbox{cos}^{-1}(\frac{\|[{\bf x}]_{\{\Omega_{j-1}\cup j+1\}}\|_{2}}{\|{\bf x}\|_{2}})}{\pi}$, respectively.

Now, we can compute the probability

[TABLE]

where $\theta_{j,1}=\frac{\mbox{sin}^{-1}(1-\frac{\sqrt{2}(\|[{\bf x}]_{\Omega\setminus 1}\|_{2}-\|[{\bf x}]_{\Omega\setminus\Omega_{j}}\|_{2})}{\|{\bf x}\|_{2}})}{\pi}$ and $\theta_{j,2}=\frac{\mbox{cos}^{-1}(\frac{\|[{\bf x}]_{\{\Omega_{j-1}\cup j+1\}}\|_{2}}{\|{\bf x}\|_{2}})}{\pi}$.

Finally, by combining (37), (38), (39), (40), (41), (42), (45), (46), (47), and (54), we obtain that

[TABLE]

Appendix C Proof of Theorem 6

Without loss of generality, assume that the first $s$ entries of ${\bf x}$ are nonzero, while the remaining $n-s$ entries are equal to zero. Therefore, we denote the support set and the non-support set of ${\bf x}$ as $\Omega=\{1,2,...,s\}$ and ${\cal N}{\cal S}=\{s+1,s+2,...,n\}$, respectively. Also, assume that the initial set of detected support indexes is $\tilde{\Omega}_{0}=\{s^{\prime}+1,s^{\prime}+2,...,s\}$, i.e., the initial set of the $s^{\prime}$ undetected support indexes is $\{1,2,...s^{\prime}-1,s^{\prime}\}$ with $|[{\bf x}]_{1}|\geq|[{\bf x}]_{2}|\geq...\geq|[{\bf x}]_{s^{\prime}-1}|\geq|[{\bf x}]_{s^{\prime}}|$, and denote $\tilde{\Omega}_{j}$ as the set of detected indexes after the first $j$ iterations.

Now, we define two events, $E_{\footnotesize{\mbox{succ}}}$: $\tilde{\Omega}_{s^{\prime}}=\Omega$ and $E_{\varnothing\rightarrow a^{\prime}}$: the order of newly detected indexes after the first $a^{\prime}$ iterations is $1\rightarrow 2\rightarrow...\rightarrow a^{\prime}$. Consider a specific event $E_{\varnothing\rightarrow s^{\prime}}$ where the order of newly detected indexes after the $s^{\prime}$ iterations is: $1\rightarrow 2\rightarrow...\rightarrow s^{\prime}$. Note that when the event $E_{\varnothing\rightarrow s^{\prime}}$ happens, the event $E_{\footnotesize{\mbox{succ}}}$ also happens. Therefore, it can be inferred that

[TABLE]

With the multiplication law of probability, $\mathbb{P}\{E_{\varnothing\rightarrow s^{\prime}}\}$ can be decomposed as

[TABLE]

where the event $E_{a^{\prime}-1\rightarrow a^{\prime}}$ is defined as: the newly detected index in the $(a^{\prime}-1)$th iteration is $a^{\prime}-1$ and the newly detected index in the $a^{\prime}$th iteration is $a^{\prime}$.

Define two events, $E_{\footnotesize{\mbox{aug}},a^{\prime}}$: the index $a^{\prime}$ is selected for support augmentation in the $a^{\prime}$th iteration and $E_{\footnotesize{\mbox{pru}},a^{\prime}}$: $\{\tilde{\Omega}_{a^{\prime}-1}\cup a^{\prime}\}\subseteq\tilde{\Omega}_{a^{\prime}}$. Then, with the multiplication law of probability, the first term and the $a^{\prime}$th term, $a^{\prime}=2,3,...,s$, on the RHS of (57) can be decomposed as

[TABLE]

[TABLE]

respectively. In the following proof, we will address the probability terms on the RHS of (58) and (59).

To begin with, consider a specific index set $\Omega^{\prime}\subset\{1,2,...,n\}$. For an analysis of the probability of the hyperplanes generated from ${\bf A}_{o}$ separating ${\bf x}$ and the estimate obtained from $\Omega^{\prime}$, $\tau_{n}({\bf A}_{r_{\Omega^{\prime}}}^{\dagger}{\bf y}_{r},\Omega^{\prime})$, we would like to compute the $\ell_{2}$-norm between both points as $\|{\bf x}-\tau_{n}({\bf A}_{r_{\Omega^{\prime}}}^{\dagger}{\bf y}_{r},\Omega^{\prime})\|_{2}$.

In a similar manner to the proof of Theorem 5 (see Appendix B), with the RIP of the matrix ${\bf A}_{r}$ and $\|{\bf e}_{r}\|_{2}\leq\frac{\|{\bf x}\|_{2}-\sqrt{2}\|[{\bf x}]_{\Omega-\{\tilde{\Omega}_{0}\cup 1\}}\|_{2}}{2+\sqrt{2}}$, it can be derived that

[TABLE]

Note that $\|{\bf x}\|_{2}-\sqrt{2}(\|[{\bf x}]_{\Omega\setminus\{\tilde{\Omega}_{0}\cup 1\}}\|_{2}-\|[{\bf x}]_{\Omega\setminus\Omega^{\prime}}\|_{2})\leq\|{\bf x}\|_{2}$ if $\{\tilde{\Omega}_{0}\cup 1\}\subset\Omega^{\prime}$.

Now, consider the step of support augmentation. In the $j$th iteration, we would like to select the index $j$ from $\Omega_{j-1}^{c}$ and obtain the augmented estimated support set ${\cal T}_{j}$.

Define two events, $E_{\tilde{\Omega}_{1},\hat{n}_{1}}$: when $j=1$, $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{1,1}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|\geq\hat{n}_{1}$ and $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{1,p^{\prime}}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|\leq\hat{n}_{1}-1,p^{\prime}\in\{\Omega_{0}\cup 1\}^{c}$, and $E_{\tilde{\Omega}_{a^{\prime}},\hat{n}_{a^{\prime}}}$: when $j=a^{\prime},a^{\prime}=2,3,...,s^{\prime}$, $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{a^{\prime},a^{\prime}}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|\geq\hat{n}_{a^{\prime}}$ and $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{a^{\prime},p^{\prime}}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|\leq\hat{n}_{a^{\prime}}-1,p^{\prime}\in\{\Omega_{a^{\prime}-1}\cup a^{\prime}\}^{c}$, conditioned on the event $E_{\varnothing\rightarrow a^{\prime}-1}$, where $\hat{n}_{j},j=1,2,...,s$ is the threshold number of support augmentation in the $j$th iteration.

Note that when the event $E_{\tilde{\Omega}_{1},\hat{n}_{1}}$ happens, the event $E_{\footnotesize{\mbox{aug}},1}$ also happens, and when the event $E_{\tilde{\Omega}_{a^{\prime}},\hat{n}_{a^{\prime}}}$ happens, the event $E_{\footnotesize{\mbox{aug}},a^{\prime}}$ conditioned on the event $E_{\varnothing\rightarrow a^{\prime}-1}$ also happens. Therefore, it can be inferred that

[TABLE]

[TABLE]

where the event $E_{\tilde{\Omega}_{j},\hat{n}_{j}}$ implies that the set of detected support indexes in the $j$th iteration would be $\tilde{\Omega}_{j}=\{\tilde{\Omega}_{0},1,2,...,j\}=\{\tilde{\Omega}_{j-1},j\}\subset{\cal T}_{j}$ ($\tilde{\Omega}_{0}=\{s^{\prime}+1,s^{\prime}+2,...,s\}$).

It can be observed that the event $E_{\tilde{\Omega}_{j},\hat{n}_{j}}$ in this step is similar to the event $E_{\Omega_{j},n_{j}}$ in the step of support detection in the proof of Theorem 5 (see Appendix B). By replacing $n_{j}$, $\kappa_{j}$, $(\|[{\bf x}]_{\Omega\setminus 1}\|_{2}-\|[{\bf x}]_{\Omega\setminus\Omega_{j}}\|_{2})$ in $\theta_{j,1}$, and $\Omega_{j-1}$ in $\theta_{j,2}$ on the RHS of (54) with $\hat{n}_{j}$, $n-s$, $(\|[{\bf x}]_{\Omega\setminus\{\tilde{\Omega}_{0}\cup 1\}}\|_{2}-\|[{\bf x}]_{\Omega\setminus\tilde{\Omega}_{j}}\|_{2})$, and $\tilde{\Omega}_{j-1}$, respectively, we obtain that

[TABLE]

where $\hat{\theta}_{j,1}=\frac{\mbox{sin}^{-1}(1-\frac{\sqrt{2}(\|[{\bf x}]_{\Omega\setminus\{\tilde{\Omega}_{0}\cup 1\}}\|_{2}-\|[{\bf x}]_{\Omega\setminus\tilde{\Omega}_{j}}\|_{2})}{\|{\bf x}\|_{2}})}{\pi}$ and $\hat{\theta}_{j,2}=\frac{\mbox{cos}^{-1}(\frac{\|[{\bf x}]_{\{\tilde{\Omega}_{j-1}\cup j+1\}}\|_{2}}{\|{\bf x}\|_{2}})}{\pi}$.

Next, consider the step of support pruning. In the $j$th iteration, we would like to obtain a subset $\Omega_{j}$, which contains $\tilde{\Omega}_{j}$ and whose cardinality is $|\Omega_{j}|=s$, of the augmented estimated support set ${\cal T}_{j}$, conditioned on the event $E_{\footnotesize{\mbox{aug}},j}$. Since $|{\cal T}_{j}|=s+1$, there are $\binom{s+1}{s}=s+1$ candidates for the selection of $\Omega_{j}$. Among them, $s-s^{\prime}+j$ candidates contain the set $\tilde{\Omega}_{j}$, and we would like to select any one of these candidates. The remaining $(s+1)-(s-s^{\prime}+j)=s^{\prime}-j+1$ candidates are not desired for the selection.

Define two events, $E_{\Omega_{1},\tilde{n}_{1}}$: when $j=1$, $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{{\cal T}}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|\geq\tilde{n}_{1},\tilde{\Omega}_{1}\subseteq{\cal T}\subset{\cal T}_{1},|{\cal T}|=s$ and $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{{\cal T}^{\prime}}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|\leq\tilde{n}_{1}-1,\tilde{\Omega}_{1}\not\subseteq{\cal T}^{\prime},{\cal T}^{\prime}\subset{\cal T}_{1},|{\cal T}^{\prime}|=s$, conditioned on the event $E_{\footnotesize{\mbox{aug}},1}$, and $E_{\Omega_{a^{\prime}},\tilde{n}_{a^{\prime}}}$: when $j=a^{\prime},a^{\prime}=2,3,...,s^{\prime}$, $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{{\cal T}}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|\geq\tilde{n}_{a^{\prime}},\tilde{\Omega}_{a^{\prime}}\subseteq{\cal T}\subset{\cal T}_{a^{\prime}},|{\cal T}|=s$ and $|\{i\mid[{\bf y}_{o}]_{i}\cdot[\hat{{\bf A}}_{{\cal T}^{\prime}}{\bf y}_{r}]_{i}\geq 0,i=1,2,...,m_{o}\}|\leq\tilde{n}_{a^{\prime}}-1,\tilde{\Omega}_{a^{\prime}}\not\subseteq{\cal T}^{\prime},{\cal T}^{\prime}\subset{\cal T}_{a^{\prime}},|{\cal T}^{\prime}|=s$, conditioned on the events $E_{\footnotesize{\mbox{aug}},a^{\prime}}$ and $E_{\varnothing\rightarrow a^{\prime}-1}$, where $\tilde{n}_{j},j=1,2,...,s$ is the threshold number of support pruning in the $j$th iteration.

Note that when the event $E_{\Omega_{1},\tilde{n}_{1}}$ happens, the event $E_{\footnotesize{\mbox{pru}},1}$ conditioned on the event $E_{\footnotesize{\mbox{aug}},1}$ also happens, and when the event $E_{\Omega_{a^{\prime}},\tilde{n}_{a^{\prime}}}$ happens, the event $E_{\footnotesize{\mbox{pru}},a^{\prime}}$ conditioned on the events $E_{\footnotesize{\mbox{aug}},a^{\prime}}$ and $E_{\varnothing\rightarrow a^{\prime}-1}$ also happens. Therefore, it can be inferred that

[TABLE]

[TABLE]

where the event $E_{\Omega_{j},\tilde{n}_{j}}$ implies that the pruned estimated support set $\Omega_{j}\supset\tilde{\Omega}_{j}$.

It can be observed that the event $E_{\Omega_{j},\tilde{n}_{j}}$ in this step is also similar to the event $E_{\Omega_{j},n_{j}}$ in the step of support detection in the proof of Theorem 5 (see Appendix B). By replacing $n_{j}$, $(\kappa_{j}-1)$, $(\|[{\bf x}]_{\Omega\setminus 1}\|_{2}-\|[{\bf x}]_{\Omega\setminus\Omega_{j}}\|_{2})$ in $\theta_{j,1}$, and $\{\Omega_{j-1}\cup j+1\}$ in $\theta_{j,2}$ on the RHS of (54) with $\tilde{n}_{j}$, $s^{\prime}-j+1$, $(\|[{\bf x}]_{\Omega\setminus\{\tilde{\Omega}_{0}\cup 1\}}\|_{2}-\|[{\bf x}]_{\Omega\setminus\tilde{\Omega}_{j}}\|_{2})$, and $\tilde{\Omega}_{j}\setminus i_{j}^{*}$ (with $i_{j}^{*}=\underset{i\in\tilde{\Omega}_{j}}{\mbox{argmin}}|[{\bf x}]_{i}|$), respectively, and multiplying $s-s^{\prime}+j$ to the second term on the RHS of (54), we obtain that

[TABLE]

where $\tilde{\theta}_{j,2}=\frac{\mbox{cos}^{-1}(\frac{\|[{\bf x}]_{\tilde{\Omega}_{j}\setminus i_{j}^{*}}\|_{2}}{\|{\bf x}\|_{2}})}{\pi}$.

Finally, by combining (56), (57), (58), (59), (61), (62), (63), (64), (65), and (66), we derive that

[TABLE]