Tree Search Network for Sparse Regression

TL;DR

The paper introduces TSN, a tree search algorithm enhanced by deep neural networks, which significantly improves sparse signal recovery performance and reduces complexity compared to existing methods.

Contribution

It presents a novel tree search algorithm driven by deep neural networks for sparse regression, outperforming existing algorithms in accuracy and efficiency.

Findings

TSN achieves lower reconstruction error than baseline methods.

TSN outperforms existing algorithms across various sensing matrices.

TSN is effective in both noiseless and noisy scenarios.

Abstract

We consider the classical sparse regression problem of recovering a sparse signal $x_0$ given a measurement vector $y = \Phi x_0+w$. We propose a tree search algorithm driven by the deep neural network for sparse regression (TSN). TSN improves the signal reconstruction performance of the deep neural network designed for sparse regression by performing a tree search with pruning. It is observed in both noiseless and noisy cases, TSN recovers synthetic and real signals with lower complexity than a conventional tree search and is superior to existing algorithms by a large margin for various types of the sensing matrix $\Phi$, widely used in sparse regression.

Figures

Tables

Table 1. TABLE I: The maximum sparsity s 0.95 subscript 𝑠 0.95 s_{0.95} of algorithm at which the exact support recovery rate is at least 95 percent (the execution time of algorithm when the sparsity | Ω | Ω |\Omega| is s 0.95 subscript 𝑠 0.95 s_{0.95} )

Alg $\setminus$ $\left(m/n\right)$	$20/100$	$20/400$	$20/800$	$20/3200$	$20/{10}^4$	$40/100$	$40/400$	$40/800$	$40/3200$	$40/{10}^4$
TSN	$\mathrm{𝟕}\left(0.7\right)$	$\mathrm{𝟓}\left(2.2\right)$	$\mathrm{𝟒}\left(2.1\right)$	$\mathrm{𝟑}\left(2.08\right)$	$\mathrm{𝟐}\left(0.8\right)$	$\mathrm{𝟏𝟕}\left(3.2\right)$	$\mathrm{𝟏𝟎}\left(2.9\right)$	$\mathrm{𝟖}\left(3.4\right)$	$\mathrm{𝟔}\left(1.9\right)$	$\mathrm{𝟓}\left(3.1\right)$
SBL	$4\left(0.5\right)$	$3\left(1.1\right)$	$2\left(1.8\right)$	$1\left(6.8\right)$	$1\left(25.7\right)$	$13\left(1.5\right)$	$7\left(2.1\right)$	$6\left(3.6\right)$	$3\left(13.1\right)$	$2\left(52.3\right)$
MMP	$4\left(0.004\right)$	$3\left(0.003\right)$	$2\left(0.002\right)$	$2\left(0.003\right)$	$1\left(0.003\right)$	$11\left(0.06\right)$	$7\left(0.03\right)$	$6\left(0.03\right)$	$4\left(0.05\right)$	$3\left(0.04\right)$
Lasso	$2\left(0.1\right)$	$2\left(0.1\right)$	$2\left(0.2\right)$	$1\left(0.1\right)$	$1\left(0.3\right)$	$9\left(0.2\right)$	$5\left(0.2\right)$	$4\left(0.2\right)$	$3\left(0.3\right)$	$3\left(0.5\right)$
gOMP	$2\left(0.003\right)$	$2\left(0.005\right)$	$2\left(0.008\right)$	$1\left(0.02\right)$	$1\left(0.08\right)$	$9\left(0.004\right)$	$5\left(0.008\right)$	$4\left(0.007\right)$	$3\left(0.04\right)$	$3\left(0.16\right)$
IHT	$1\left(0.001\right)$	$1\left(0.001\right)$	$1\left(0.001\right)$	$1\left(0.004\right)$	$1\left(0.008\right)$	$6\left(0.006\right)$	$4\left(0.007\right)$	$3\left(0.007\right)$	$2\left(0.01\right)$	$2\left(0.03\right)$
GFLSTM	$4\left(0.02\right)$	$2\left(0.02\right)$	$2\left(0.02\right)$	$1\left(0.02\right)$	$1\left(0.02\right)$	$11\left(0.02\right)$	$6\left(0.02\right)$	$4\left(0.02\right)$	$2\left(0.02\right)$	$1\left(0.02\right)$
LVAMP	$3\left(0.02\right)$	$2\left(0.02\right)$	$1\left(0.02\right)$	$1\left(0.02\right)$	$1\left(0.02\right)$	$11\left(0.02\right)$	$2\left(0.02\right)$	$1\left(0.02\right)$	$1\left(0.02\right)$	$1\left(0.02\right)$

Table 2. TABLE II: Tuple ( u , b , t ) 𝑢 𝑏 𝑡 (u,b,t) of each algorithm under NOMA with 10 10 10 active users. The tuple contains perfect recovery rate u 𝑢 u ( 0 ≤ u ≤ 1 0 𝑢 1 0\leq u\leq 1 ) for all active users, block error rate b 𝑏 b ( 0 ≤ b ≤ 1 0 𝑏 1 0\leq b\leq 1 ), and execution time t 𝑡 t (s)

Alg. $\setminus$ E${}_{\text{b}}$/N0	0 (dB)	20 (dB)
TSN	$(0.93,0.01,3.1)$	$(0.98,0.002,3.7)$
GFLSTM	$(0.36,0.1,0.03)$	$(0.45,0.1,0.03)$
SBL	$(0.07,0.22,3.2)$	$(0.27,0.21,3.1)$
MMP1	$(0.18,0.28,3.8)$	$(0.21,0.27,3.8)$
MMP2	$(0.18,0.28,37.6)$	$(0.21,0.27,39.2)$
Lasso	$(0.15,0.2,0.2)$	$(0.16,0.2,0.2)$
IHT	$(0.06,0.24,0.11)$	$(0.1,0.23,0.11)$
gOMP	$(0.01,0.4,0.06)$	$(0.01,0.4,0.05)$

Table 3. TABLE III: Performance for MNIST image recovery in terms of signal error and signal recovery rate when SNR is 25 25 25 dB and sparsity of the target MNIST image is 215 215 215

Algorithm	$\Vert x_0-\hat{x}\Vert /\Vert x_0\Vert$	$\left|\hat{\Omega }\cap \Omega \right|/\left|\Omega \right|$	Running time
TSN	$0.1301$	$0.8577$	11.47
GFLSTM	0.2605	0.7805	0.091
LVAMP	0.7419	0.5493	1.457
SBL	0.7488	0.5513	116.3
MMP	1.0178	0.4884	391.9
Lasso	0.4684	0.7342	2.622
IHT	0.6452	0.6422	0.267
gOMP	1.0627	0.4403	6.948
SP	0.8126	0.5028	0.233
CoSaMP	-	0.4561	0.064

Table 4. TABLE IV: Performance for OMNIGLOT image recovery in terms of signal error and signal recovery rate when SNR is 25 25 25 dB and sparsity of the target OMNIGLOT image is 160 160 160

Algorithm	$\Vert x_0-\hat{x}\Vert /\Vert x_0\Vert$	$\left|\hat{\Omega }\cap \Omega \right|/\left|\Omega \right|$	Running time
TSN	$0.1741$	$0.9256$	2.106
GFLSTM	0.7008	0.6289	0.054
LVAMP	0.7654	0.6039	0.855
SBL	0.7665	0.6359	197.1
MMP	0.9506	0.5811	265.1
Lasso	0.4082	0.8557	4.205
IHT	0.6961	0.6991	0.969
gOMP	1.0096	0.4475	11.49
SP	0.7732	0.6811	1.277
CoSaMP	-	0.3225	0.552

Table 5. TABLE V: Probability where the true support Ω Ω \Omega is equal to the top | Ω | Ω |\Omega| indices of the trained DNN-SR output when SNR = 5 5 5 dB

Data type $\setminus$ $\left|\Omega \right|$	$1$	$2$	$3$	$4$	$5$	$6$
Noisy data	$1.00$	$1.00$	$0.98$	$0.94$	$0.76$	$0.45$
Noiseless data	$0.92$	$0.81$	$0.75$	$0.71$	$0.56$	$0.31$

Table 6. TABLE VI: Maximum sparsity s 0.95 subscript 𝑠 0.95 s_{0.95} of each evaluated algorithm with exact support recovery rate above 95% and execution time of each algorithm for sparsity | Ω | Ω |\Omega| being s 0.95 subscript 𝑠 0.95 s_{0.95}

Alg $\setminus$ $\left(m/n\right)$	$500/{10}^4$	$1000/{10}^4$	$1500/{10}^4$
TSN	$\mathrm{𝟗𝟎}\left(6.4\right)$	$\mathrm{𝟐𝟏𝟕}\left(18.9\right)$	$\mathrm{𝟑𝟓𝟏}\left(29.8\right)$
SBL	$18\left(128.3\right)$	$57\left(220\right)$	$280\left(373.5\right)$
MMP	$75\left(21.8\right)$	$166\left(28.4\right)$	$239\left(453.4\right)$
Lasso	$65\left(7.2\right)$	$215\left(19.6\right)$	$275\left(24.3\right)$

Equations

$$\underset{\tilde{x}\in {\mathbb{K}}^n}{\min }\Vert P_{\mathcal{R}({\Phi }_{\Gamma })}^{\perp }y-{\Phi }_D{\tilde{x}}^D\Vert =0,$$

$${\mathbf{\tau }}_1:(q,z,ϵ,\mathbf{l},\mathbf{g},t_{\text{max}})=(m,1,\overline{ϵ},(3,1),(60,1),\infty )$$

$${\mathbf{\tau }}_2:(q,z,ϵ,\mathbf{l},\mathbf{g},t_{\text{max}})=(m,1,\overline{ϵ},(2,1{)}^2,(60,1{)}^2,\infty )$$

$${\mathbf{\tau }}_3:(q,z,ϵ,\mathbf{l},\mathbf{g},t_{\text{max}})=(m,1,\overline{ϵ},(2,1{)}^2,(60,1{)}^2,5)$$

$$p_{x_0}(x\mid \bar{s}):=\frac{p_{x_0}(x)\cdot 1(|\mathrm{supp}(x)|=\bar{s})}{{\sum }_{\tilde{x}\in {\mathbb{K}}^n\text{ s.t. }|\mathrm{supp}(\tilde{x})|=\bar{s}}{p}_{x_0}(\tilde{x})},$$

$$L(\theta ):={\mathbb{E}}_{(y,x)}[\text{ce(}{f}_{\theta }(y),{\text{u}}_n(\mathrm{supp}(x)))],$$

$$\Vert P_{\mathcal{R}({\Phi }_{\Gamma })}^{\perp }y-{\Phi }_H{\tilde{x}}^H\Vert =0$$

$$(\gamma ,\eta )=\underset{\bar{\gamma }\in {\mathbb{R}}^a,\bar{\eta }\in \mathbb{R}}{\mathrm{argmin}}L(\bar{\Phi },y):=\log |\Sigma |+y^{\top }{\Sigma }^{-1}y,$$

$$y=\Phi \bar{x}+w\text{ and }\mathrm{supp}(\bar{x})\subseteq {\Psi }_t$$

$$E:={\cup }_{J\subseteq \{1:n\}\text{ s.t. }|J\cap (\Omega \setminus \Gamma )|<|\Omega \setminus \Gamma |\text{ and }|J|<m}\mathcal{R}({\Phi }_J).$$

$$\mathbf{\tau }:(q,z,ϵ,\mathbf{l},\mathbf{g},t_{\text{max}})=(200,110,\overline{ϵ},(1,1),(1,1),\infty ),$$

Full text

I Introduction

The sparse linear regression (SR), referred to as compressed sensing (CS) [1], has received much attention in many machine learning and signal processing applications111Some examples are feature selection [2, 3], signal reconstruction [4, 5, 6], denoising [7], and super-resolution imaging [8, 9].. Its goal is to recover $k$-sparse222$x_{0}$ is $k$-sparse if it has at most $k$ nonzero elements. signal vector $x_{0}\in\mathbb{K}^{n}$ and its support $\Omega$ from under-sampled measurement vector $y\in\mathbb{K}^{m}$ such that $m\leq n$ and $y=\Phi x_{0}+w$, where $\Omega$ denotes the set of nonzero elements in $x_{0}$, $\Phi\in\mathbb{K}^{m\times n}$ is a known sensing matrix, and $w\in\mathbb{K}^{m}$ is the noise vector.

I-A Existing deep neural network for SR

Deep neural networks (DNNs) have contributed to notable performance improvements in fields such as image processing [10, 11], natural language processing [12], and reinforcement learning [13]. Consequently, intense research has been devoted to the development of a tailored DNN for SR (DNN-SR) to estimate sparse signals; it outputs an estimate of $k$-sparse signal $x_{0}$ or its support $\Omega$ from measurement vector $y$. Gregor and LeCun proposed a DNN-SR structure by observing that the iterations in each layer of a feed-forward neural network can represent the update step in an existing SR algorithm called iterative shrinkage and thresholding (ISTA) [14]. Subsequent studies addressed variants of the DNN-SR architecture based on the unfolding process in the context of this algorithm to improve performance by incorporating a nonlinear activation function [15, 16], reducing the training complexity using shared parameters over the DNN layers [17], and developing a structured sparse and low-rank model [18]. In addition, there are some recent works to study theoretical properties of learned variants of ISTA [19, 20, 21]. Similarly, learned variants of approximate message passing (AMP) for SR have been studied [22, 23, 6] by exploiting the unfolding process. On the other hand, a DNN-SR structure based on the alternating direction method of multipliers was proposed for generating a transform matrix to filter magnetic resonance imaging data [24, 25] and a DNN-SR architecture based on a generative model has been proposed [26, 27]. Besides, the correlation among the different sparse vectors was used in the DNN-SR proposed in [28, 29] when multiple measurement vectors share a common support. Lately, He et al. suggested [30] that the update step for sparse Bayesian learning (SBL) [31] can be formed into a gated feedback long short-term memory (GFLSTM) network, which is a widely used recurrent DNN structure [32]. This DNN-SR structure showed a comparable performance to that of SBL in the case when the sensing matrix consists of columns with high correlation.

I-B Scope and contribution

As the studies mentioned in Section I-A showed, DNN can be exploited to solve the SR problem and demonstrate potential to outperform existing SR algorithms. However, there is not enough research showing that DNN-SR is not limited to image processing and enables uniform recovery of synthesis sparse signals with better performance than existing SR methods. On the other hand, deep learning combined with optimization techniques based on tree search has shown better performance than existing methods without tree search. Typical examples include AlphaGo [33], which applies Monte Carlo tree search to deep reinforcement learning, and a deep reinforcement neural network [34] trained by data generated from an offline Monte Carlo tree search to outperform deep Q-networks. Motivated by these works, we first propose a tree search algorithm driven by deep neural network for SR (TSN) to improve the performance of DNN-SR.333Once the support is determined, the problem of estimating $x_{0}$ reduces to a standard overdetermined linear inverse problem, which can be easily solved. Therefore, we focus on a type of DNN-SR that recovers the true support $\Omega$, such that it takes $y$ as its input and outputs a probability vector $v:=(v_{1},...,v_{n})$ whose $|\Omega|$-largest indices represent the estimate of $\Omega$. TSN performs a tree search to find support $\Omega$ of $x_{0}$ based on a trained DNN-SR of the abovementioned type.

Note that the tree search in TSN is applied to DNN-SR as a post-processing framework. That is, we trained a single DNN-SR independent of the tree search and used the single network to generate all the nodes in the search tree in TSN. TSN has the following three main features, i.e., DNN-based index selection for tree search and pruning the tree.

• •

(The DNN-based index selection for tree search) In the conventional tree search, e.g., multipath matching pursuit (MMP) [35], to estimate the support, each parent node representing a partial support estimate generates its child nodes through the index selection based on orthogonal matching pursuit (OMP): selecting indices according to the largest correlations measured by the inner product with the residual vector. In this study, we consider the DNN-based index selection in addition to the OMP-based index selection for generating child nodes. Figure 1 shows that the percentage of true indices among the support estimate of size $v$, obtained by the trained DNN-SR, is higher than those using other conventional SR methods. This implies that a true index can be included in the child nodes with a higher probability by using the DNN-based index selection than the OMP-based approach. He et al. showed in [30] that the percentage of true indices among the m-largest predicted DNN outputs (the loose accuracy) is significantly higher than those obtained by using other SR methods. This observation also supports our argument. Thus, given that the DNN-based tree search can find the support with a few child nodes, combining DNN and tree search can improve the performance over using them separately.

• •

(Pruning the search tree) Suppose that $q$ child nodes are generated at each parent node of the search tree. Then, the number of nodes in the tree exponentially increases with the tree depth $d$ to approximately $\sum_{i=0}^{d-1}q^{i}$. Hence, to reduce the complexity of the tree search, we propose a pruning method to remove leaf nodes at certain depths, except for those showing the smallest signal errors. For instance, preserving one node at every depth $t\ll d$ through the pruning method, the number of nodes in the resulting tree with depth $d$ is approximately $(d/t)\cdot(\sum_{i=0}^{t-1}q^{i})$. Given that the maximum exponent of $q$ decreases from $d$ to a constant $t$, the searching complexity is efficiently reduced, and it is linearly dependent on depth $d$.

• •

(Generating the extended support estimation $\Psi$ of size $m-1$ via DNN-SR) Note that each node in the search tree in TSN has a partial support estimate $\Delta$ generated from its parent node via the DNN-based index selection. Then every node in TSN generates a support estimate of size $k$ by utilizing its parital support estimate $\Delta$ and the trained DNN-SR. To do this, each node generates an extended support estimate $\Psi$ of size $m-1$ via the DNN-based index selection at the first stage, and estimate the support by selecting $k$ indices out of the set at the second stage. This two-stage process is a variant of the two-stage process shown in [36]; the existing process selects an extended support estimate $\Psi$ of $m-1$ indices based on OMP, whereas the proposed process is based on DNN.

It is guaranteed theoretically and experimentally in [36] that exploiting the extended support estimate $\Psi$ of size $m-1$, generated by the OMP-based rule, improves the performance for the support recovery in comparison to the case without considering it, i.e., the case when the size of $\Psi$ is equal to $k$.444[36] shows that this re-estimation method using an extended support estimate of size $m-1$ is guaranteed to further mitigate the sufficient conditions for SR algorithms based on the OMP-based index selection to restore the sparse signal. We accepted this principle, i.e., selecting the $m-1$ incides instead of $k$ incides, for estimating the support at each node, but used a different index selection technique, i.e., the DNN-based index selection. As shown in Figure 1 and experimental results in [30], the DNN-based index selection has shown better performance for the loose accuracy and lower complexity than OMP-based approaches. This supports our claim that combining DNN and the extended support estimation of size $m-1$ improves the recovery performance for the sparse signal. We demonstrated it in Section IV-A.

We utilize the GFLSTM [30] or learned vector AMP (LVAMP) [23] as the DNN-SR used in TSN to demonstrate that TSN improves the performance of a typical DNN-SR. Experimental results also suggest that TSN significantly improves the recovery performance of DNN-SR, has a lower complexity than the conventional tree search, and outperforms existing SR algorithms in both noiseless and noisy cases with various types of the sensing matrix $\Phi$. For example, the maximal sparsity (i.e., the maximal size $|\Omega|$ of target support) to uniformly recover $x_{0}$ using TSN is two times larger than those using SBL and the GFLSTM network from simulations with a Gaussian sensing matrix of dimension $20\times 100$ and noiseless measurements. These tests were based on all synthesis sparse signals. It suggests that TSN can improve performance in various domains using SR. In this paper, to provide its examples, we evaluated performance of nonorthogonal multiple access (NOMA) in communication system and image restoration without using image training data, and showed the superiority of TSN.

II Notation

$\mathbb{K}$ denotes the real $\mathbb{R}$ or complex $\mathbb{C}$ field and $\mathbb{N}$ denotes the set of natural numbers. The set $\{i,i+1,...,j\}$ is denoted by $\{i:j\}$. For a matrix $A:=[a_{1},...,a_{n}]\in\mathbb{K}^{m\times n}$, submatrices of $A$ with columns indexed by $J\subseteq\{1:n\}$ and rows indexed by $Q\subseteq\{1:m\}$ are denoted by $A_{J}$ and $A^{Q}$, respectively. $\mathcal{R}(A)$ denotes the range space spanned by the columns of $A$. $A^{\top}$ ($A^{*}$) denotes the (Hermitian) transpose of $A$. $P^{\perp}_{\mathcal{R}(A)}$ denotes the projection matrix onto the orthogonal complement of $\mathcal{R}(A)$. For a vector $v:=(v_{1},...,v_{n})\in\mathbb{K}^{n}$, $\operatorname{abs}(v)$ denotes a vector $z$ whose element $z_{i}$ is the absolute value of $v_{i}$ and the Frobenius norm of $v$ is denoted by $\left\|v\right\|$. $\operatorname{supp}(v)$, the support of $v$, denotes the index set of nonzero elements in $v$. $T_{l}(v)$ is the operator whose output is an index set of size $l$ with the $l$-largest absolute values in the input vector $v:=(v_{1},...,v_{n})$.

III High-level description of TSN

The propposed TSN recovers the target signal $x_{0}$ and its support $\Omega$ given $y$ and $\Phi$ by utilizing a trained DNN-SR defined by function $f_{\theta}(\cdot):\mathbb{K}^{m}\rightarrow\mathbb{T}^{n}$ as its input, where $\theta$ is the set of training parameters in the network and $\mathbb{T}^{n}$ represents $(n-1)$-dimensional probability simplex $\{v\in\mathbb{R}^{n}|v_{1}+\dots+v_{n}=1\textup{ and }v_{i}\geq 0\textup{ for }i\in\{1:n\}\}$. For sparse vector $z\in\mathbb{K}^{n}$ and measurement vector $h=\Phi z\in\mathbb{K}^{m}$, given sensing matrix $\Phi$, function $f_{\theta}(\cdot)$ takes vector $h$ as its input and is trained to return vector $v=(v_{1},...,v_{n})=f_{\theta}(h)\in\mathbb{T}^{n}$ such that $v_{i}=1/|\operatorname{supp}(z)|$ for $i\in\operatorname{supp}(z)$ and $v_{i}=0$ for $i\notin\operatorname{supp}(z)$. Each element $v_{i}$ of vector $v$ indicates the probability that index $i$ belongs to the support of $z$. For instance, if $n$ is 4 and the support of $z$ is $\{1,3\}$, function $f_{\theta}(\cdot)$ is trained to return output vector $f_{\theta}(h)$ equal to $(1/2,0,1/2,0)$. The detailed process to train DNN $f_{\theta}(\cdot)$ used in TSN and its effect are shown in Appendix 1.

In the rest of this section, we introduce main features of TSN to estimate the support $\Omega$. The detailed process of TSN is shown in Algorithm 5 in Appendix B.

III-A The DNN-based index selection in TSN

Suppose that there exists a partial support estimate $\Delta$ of the target signal $x_{0}$, which corresponds to a parent node of the tree. Then, to find the remaining support $\Omega\setminus\Delta$ outside $\Delta$, we expand a branch from the node by generating multiple partial support estimates (child nodes) $\{\Delta_{1},\Delta_{2},...,\Delta_{q}\}$ given $\Delta$, where $q$ represents the number of child nodes. Each index set $\Delta_{i}=h_{i}\cup\Delta$ for $i\in\{1:q\}$ is obtained by adding element $h_{i}$ to $\Delta$ where $\Theta:=\{h_{1},...,h_{q}\}$ is an estimate of the remaining support $\Omega\setminus\Delta$ with size $q$. This estimate is obtained by considering the residual vector $P^{\perp}_{\mathcal{R}(\Phi_{\Delta})}y$, the orthogonal complement of the columns in $\Phi$, indexed by $\Delta$ onto $y$, as the trained DNN-SR input and selecting the $q$-largest elements $T_{q}(f_{\theta}(P^{\perp}_{\mathcal{R}(\Phi_{\Delta})}y))$ of its output. Figure 2 shows an example of the DNN-based index selection in TSN compared to the conventional tree search for SR, e.g., MMP, when $q$ is set to $2$. Suppose that the parent node $\Delta$ is $\{5\}$ at the tree depth $1$ of MMP in Figure 2. Then, MMP generates $q\,(=2)$ child nodes as $\{1,5\}$ and $\{2,5\}$ by using the OMP-based index selection $T_{q}(\Phi^{*}P^{\perp}_{\mathcal{R}(\Phi_{\Delta})}y)=\{1,2\}$. Conversely, TSN generates $q$ child nodes as $\{2,4\}$ and $\{4,5\}$ by using the DNN-based index selection $T_{q}(f_{\theta}(P^{\perp}_{\mathcal{R}(\Phi_{\Delta})}y))=\{2,5\}$ when the parent node $\Delta$ is $\{4\}$.

Note that the DNN-based index selection uses one common DNN-SR $f_{\theta}(P^{\perp}_{\mathcal{R}(\Phi_{\Delta})}y)$ to create child nodes from each parent node with a different partial support estimate $\Delta$. In order for this trained DNN-SR to provide the remaining support $\Omega\setminus\Delta$ irrespective of the set $\Delta$ given at each parent node, we learn the DNN-SR $f_{\theta}(\cdot)$ by using Algorithm 1 such that the following condition (1) is satisfied for any index set $\Gamma\subseteq\{1:n\}$

[TABLE]

where $D:=T_{q}(f_{\theta}(P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y))$ is the index set of size $q$, obtained by the DNN-based index selection. Given that $D$ satisfying condition (1) where $\Gamma=\Delta$ includes the true remaining support $\Omega\setminus\Delta$ from Lemma III.1, therefore, a true remaining index, which is not in the parent node, is added to $|\Omega\setminus\Delta|$ of its $q$ child nodes if the DNN-SR is ideally trained. The proof of Lemma III.1 is shown in Appendix C.

Lemma III.1

Suppose that $|\Omega|<m$ and every $m$ columns in $\Phi$ exhibit full rank. For any pair $(D,\Gamma)$ of index sets in $\{1:n\}$ satisfying (1) such that $|D|<m$, if $P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y$ is uniformly sampled from $\mathcal{R}(\Phi_{\Omega\setminus\Gamma})$, the index set $D$ includes $\Omega\setminus\Gamma$ ($D\supseteq\Omega\setminus\Gamma$) almost surely.

III-B Pruning the search tree in TSN

TSN prunes the search tree to reduce its complexity, as its example is shown in Figures 2 and 3. In Figure 2, pruning the tree in TSN is executed by remaining $g$ ($=$1) nodes $(\Delta_{2})$ having the minimum signal errors among all nodes $\Delta_{1}:=\{1,2\},\Delta_{2}:=\{1,7\},\Delta_{3}:=\{2,4\},\Delta_{4}:=\{4,5\}$ given at tree depth $2$. This process consists of the following three steps, which are illustrated in Figure 3.555In Figure 3, an input $z$ of TSN, detailed in Appendix B, is set to 1.

In the first step, an extended support estimate $\Psi_{i}:=\Delta\cup T_{m-1-|\Delta|}(f_{\theta}(P^{\perp}_{\mathcal{R}(\Phi_{\Delta})}y))$ of size $m-1$ is generated at each node $\Delta_{i}$ by using the trained DNN-SR and the residual vector $P^{\perp}_{\mathcal{R}(\Phi_{\Delta})}y$. In the second step, a $k$-support estimate666The $k$-support, $\Omega(k)$, of $x_{0}$ denotes any index set satisfying $\Omega(k)\supseteq\Omega$ and $|\Omega(k)|=k$. $\bar{\Omega}_{i}$ is obtained by selecting $k$ indices from each extended support estimate $\Psi_{i}$ through the ridge regression suggested in [31]. A detailed description of the ridge regression is shown in Appendix B-A2. In the noiseless case ($w=0$), given that this ridge regression is equal to the least-squares regression, $\bar{\Omega}_{i}$ is obtained by $T_{k}(\underset{x\in\mathbb{K}^{n}}{\arg\min}\left\|\Phi_{\Psi_{i}}x^{\Psi_{i}}-y\right\|)$. In the third step, the signal error of each node is calculated as the residual norm $\left\|P^{\perp}_{\mathcal{R}(\Phi_{\bar{\Omega}_{i}})}y\right\|$ by using the $k$-support estimate $\bar{\Omega}_{i}$, and then $g$ nodes having the minimum errors are selected.

The least-squares method ($\underset{x\in\mathbb{K}^{n}}{\arg\min}\left\|\Phi_{\Psi_{i}}x^{\Psi_{i}}-y\right\|$) provides $x_{0}$ as the unique solution in the noiseless case if $\Psi_{i}\supseteq\Omega$, $|\Psi_{i}|<m$, and $\Phi_{\Psi_{i}}$ has full column rank. Note that the probability of satisfying $\Psi_{i}\supseteq\Omega$ increases with the dimension of $\Psi_{i}$. In addition, if $\Psi_{i}\supseteq\Omega$ holds, the inversion problem of SR can be simplified as a problem where $\Phi$ is replaced by submatrix $\Phi_{\Psi_{i}}$. Therefore, we set the dimension of extended support estimate $\Psi_{i}$ to $m-1$; a further detailed information is shown in Appendix B-A2.

If the extended support estimate $\Psi_{i}$ of $m-1$ is obtained by the OMP-based index selection shown in [36], then the first and second steps correspond to an existing SR algorithm called two-stage orthogonal subspace matching pursuit with sparse Bayesian learning (TSML). Therefore, these two steps can be interpreted as a modification of TSML using DNN. The experimental results in Section IV-A show that this modification reduces the complexity of TSN and improves performance, thus demonstrating its validity.

IV Numerical experiments

In this section, we verify the performance of TSN against some conventional SR algorithms, namely, generalized orthogonal matching pursuit (gOMP) [37]777 Three indices were selected per iteration in gOMP., compressive sampling matched pursuit (CoSaMP) [38], subspace pursuit (SP) [39], iterative hard thresholding (IHT) [40], MMP [35]888MMP1 and MMP2 shown in this section are the MMP depth first (MMP-DF) algorithms [35] such that the number $N_{\textup{max}}$ of path candidates are $L^{s}$ and $500$, respectively, with the same expansion number $L$ equal to $4$. Thus, MMP1 is the MMP-DF with full tree search., SBL [31, 41], and basis pursuit denoising (Lasso) [42]. We also compared TSN to some state of the art DNN-based SR algorithms, namely, GFLSTM [30], a learned variant of ISTA (LISTA) [14], learned AMP (LAMP) [23], and LVAMP [23]. We set the DNN-SR used in TSN to GFLSTM to evaluate that TSN improves the performance of its target DNN-SR.

IV-A Real-valued case

IV-A1 Experimental settings

Let $\mathcal{N}(a,b)$ denote the real Gaussian distribution with mean $a$ and variance $b$. We sample $\Phi\in\mathbb{K}^{m\times n}$ such that its elements independently follow $\mathcal{N}(0,1)$ and its columns are $l_{2}$-normalized. Support $\Omega$ of $x_{0}$ is generated from a uniform distribution, and signal vector $x_{0}$ is obtained such that each of its nonzero elements is independently and uniformly sampled from $-1$ to $1$, excluding the interval from $-$0.1 to 0.1. To consider noisy signals, the average signal-to-noise ratio (SNR) per sample ($\textup{SNR}:=\mathbb{E}\left\|\Phi x_{0}\right\|^{2}/\,\mathbb{E}\left\|w\right\|^{2}$) is defined as the ratio between the power of the measured signal and that of noise. Each element of the noise vector $w$ follows $\mathcal{N}(0,\sigma^{2}_{w})$, where $\sigma_{w}$ is dependent on the given SNR environment.

For the DNN-SR $f_{\theta}(\cdot)$ used in TSN, we used GFLSTM network for sparse regression, as suggested in [30], where the hidden unit size, the number of unfolding steps, and the layer size are set to $425$, $11$, and $2$, respectively. Further details on this network are provided in [30]. To train the GFLSTM $f_{\theta}(\cdot)$, we used Algorithm 1 in Appendix 1 whose input $(k_{1},k_{2},s_{d},s_{b},n_{e})$ and v_{\textup{SNR{}_{\textup{dB}}}} are set to $(1,10,6\cdot 10^{5},250,400)$ and a fixed SNR in decibels (dB), respectively. We used RMSprop optimization with learning rate $\eta_{i}$ of $0.001$ for epoch $i\leq 250$ and $0.001/4^{j}$ for epoch $i$ from $201+50j$ to $250+50j$ ($j\in\{1:3\}$).

We varied sparsity $s:=|\Omega|$ of $x_{0}$ from $1$ to $9$ and set input value $k$ in TSN to $9$; the true sparsity $s$ was not given but only its maximal value of 9 was given to TSN, whereas the sparsity $s$ was given to the other evaluation algorithms. We evaluated the proposed TSN (Algorithm 5 in Appendix B) with three examples, namely, TSNi $:=$ TSN($y,\Phi,k,\boldsymbol{\tau}_{i}$), where the following set $\boldsymbol{\tau}_{i}$ of input parameters of TSNi is used for $i\in\{1:3\}$:

[TABLE]

where \bar{\epsilon}:=\max[(\left\|y\right\|\cdot 10^{-v_{\textup{SNR{}_{\textup{dB}}}}/20}),10^{-5}] is the signal error bound, $t_{\textup{max}}$ is the maximum running time of TSN in second, $(\boldsymbol{l},\boldsymbol{g})$ determines how many nodes are left at particular tree depths via the pruning method (Appendix B), and $(a,b)^{2}:=(a,b,a,b)$.

The performance was evaluated according to two metrics, namely, the expected value of the normalized distance between $x_{0}$ and its estimate $\hat{x}$ obtained from the evaluated algorithm, $\mathbb{E}(\left\|x_{0}-\hat{x}\right\|/\left\|x_{0}\right\|)$, and the probability where the error between the target signal $x_{0}$ and its estimate $\hat{x}$ is smaller than the noise magnitude, i.e., $\mathbb{P}(\left\|\Phi\hat{x}-\Phi x_{0}\right\|\leq\left\|w\right\|)$.999The second metric $\mathbb{P}(\left\|\Phi\hat{x}-\Phi x_{0}\right\|\leq\left\|w\right\|)$ in the noiseless case is equal to the rate of successful signal recovery, $\mathbb{P}(x_{0}=\hat{x})$, in our simulation setting, as every set of $s$ columns of $\Phi$ has the rank $s$ for $s\leq m/2$.

IV-A2 Performance evaluation

Figures 4(a) and (b) show the performance of a modified TSN that replaces the DNN-based index selection with the OMP-based index selection in terms of signal recovery rate and execution time. The proposed TSN and its variant are denoted by TSN (GFLSTM) and TSN (OMP), respectively, in Figures 4(a) and (b). The modified TSN using the OMP-based index selection outperforms MMP at a comparable complexity, and thus the proposed tree search, even omitting the DNN, is superior to the existing tree search. Still, the figures show that TSN with the DNN-based selection has a lower complexity and outperforms its modified one using the OMP-based selection, thus confirming the effectiveness of jointly exploiting tree search and DNN-based index selection. We also compared TSN to other widely used DNN-based SR algorithms, including LISTA, LAMP, and LVAMP, obtaining the results shown in Figures 4(c) and (d). LVAMP outperforms both LISTA and LAMP and retrieves a comparable performance to GFLSTM. TSN uniformly improves the performance of GFLSTM and LVAMP by setting the DNN used in TSN to GFLSTM and LVAMP, respectively, and TSN using GFLSTM has a lower complexity and outperforms TSN using LVAMP. In addition, TSNi shows better performance than TSNj, whereas the complexity of TSNi is larger than that of TSNj for $i,j$ in $\{1:3\}$ such that $i<j$.101010The total number of nodes in the search tree of TSNj is smaller than that of TSNi. Given that the performance does not notably improve when the number of layers in LISTA, LAMP, and LVAMP is larger than 12, we considered 12 layers.

Figures 5(a) and (b) show the performance comparison to conventional SR algorithms without using DNN in terms of signal recovery rate and signal error, respectively, whereas Figure 5(c) shows the algorithm execution time111111We simulated TSN and GFLSTM by using TensorFlow running on an E5-2640 v4 2.4GHz CPU endowed with a Nvidia Titan X GPU; the other algorithms were implemented in MATLAB., in the noiseless case. The results in Figures 5(a) and (b) show that TSN exhibits the best recovery performance of the signal $x_{0}$ and its support $\Omega$ in the whole sparsity region for the noiseless case.121212We observe that MMP1 does not improve the performance of MMP2 though the tree of MMP1 is extended from that of MMP2. In particular, it is observed that the maximum sparsity $s$ of $x_{0}$ for TSN to uniformly recover $x_{0}$ is 6 or 7, which is larger than two times those for other algorithms, by measuring the maximum sparsity with the signal error below $10^{-10}$ in Figure 5(b). Figure 5(c) shows that TSN algorithms depicted in Figures 5(a) and (b) have execution times under 1 second for the true sparsity of up to $6$, which is around their phase-transition points shown in Figure 5(b).

The performance is compared in the noisy cases in Figures 5(d)–(h). These results show that TSN outperforms other algorithms to recover $x_{0}$ for most sparsity region even in the noisy case. Note that from Figures 5(d) and (e), there exists a tradeoff between the recovery performance and complexity, as similarly shown in Figures 5(a)–(c), of TSNi for $i\in\{1:3\}$ in the case where SNR is $25$ dB.131313As $\mathbb{P}(\left\|\Phi\hat{x}-\Phi x_{0}\right\|\leq\left\|w\right\|)$ when SNR is $25$ dB is observed similarly with Figure 5(a), we omitted to show this plot. It is observed that the execution time of TSN3 is under 2 seconds for sparsity $s\leq 9$ in Figure 5(e). Figures 5(f)–(h) show that in low SNR case ($5$ dB), TSN3 takes the smallest execution time, under $1$ second in the whole sparsity region, with a similar recovery performance of $x_{0}$ among TSNi for $i\in\{1:3\}$. The smaller SNR, the easier it is to find a $k$-sparse vector $z$ such that $\left\|\Phi z-\Phi x_{0}\right\|\leq\left\|w\right\|$. For that reason, TSN3, whose nodes in the tree are fewer than those of TSN1 and TSN2, is sufficient for providing a signal estimate existing within the noise bound $\left\|w\right\|$ in low SNR case, as observed in Figures 5(f)–(h).

IV-B Complex-valued case

Even if $x_{0}$ and $\Phi$ are complex, we observed that TSN improved its target DNN-SR and outperformed other SR algorithms with the same tendency as in Figure 5 in both noiseless and noisy cases. We set $\Phi$ to a complex Gaussian matrix, a partial DFT matrix, and a complex matrix with highly correlated columns [30, 43], which have been typically used in the CS application. Other DNN-SR algorithms except for TSN and GFLSTM, shown in Figure 4, can not be extended to the complex-valued case, they are omitted from the performance comparison result. Figure 6 demonstrates our argument and shows that in the noiseless case TSN almost achieves the ideal limit [44] of sparsity $(s=10)$ for the uniform recovery of $x_{0}$ regardless of the type of $\Phi$. Details including the setting parameters, the noisy case (SNR = $25$ dB), and the complexity are shown in Appendix D.

IV-C Scalability

We evaluated the scaling of performance and complexity in the TSN at different sizes of sensing matrix $\Phi$. Table I lists the maximum sparsity and execution time of each algorithm at which the recovery rate is at least 95%. TSN notably outperforms the other methods even when $(m,n)$ increases beyond $(20,100)$, and its performance and complexity scale well with $n$.141414The execution time for TSN sometimes decreases with $n$ following the trend of $s_{0.95}$. The setting parameters are detailed in Appendix E.

We observed that as $m$ approaches $n$ and $m$ increases, the complexity versus performance of TSN relatively increases. It implies that there exists a trade-off between performance and computational complexity of TSN in the case when $m$ is large enough. To show this, results with matrix size $m\times n$ reaching $1500\times 10^{4}$ are detailed in Appendix E. Thus, at least for SR problems with a small value of $m$, regardless of the value of $n$, we can conclude that TSN has a higher performance versus complexity than other SR algorithms. Note that the smaller the sampling rate $m/n$, the more difficult it is to recover the signal in the SR problem. Our study suggests that TSN enables to derive the state-of-the-art result to solve the SR problem for this difficult case.

IV-D Estimation for structured sparse signal

TSN can further improve its recovery performance when the signal distribution $p_{x_{0}}(\cdot)$ has an additional structure as TSN is based on DNN-SR trained by data sampled from that distribution. To provide an example (Figure 7), we added a condition to the signal distribution used in Sections IV-A and IV-B, such that all nonzero elements of $x_{0}$ are positive. Figure 7(a) shows the performance improvement of TSN compared to Figure 5(a). It is observed in Figure 7(a) that TSN almost achieves the ideal limit of sparsity $(s=10)$ for the uniform recovery of $x_{0}$, while the maximum sparsity for perfect signal recovery through SBL is 3; TSN performs nearly three times better than SBL. Similarly, with the real case, the performance improvement of TSN is observed in the complex case when the real and imaginary parts of nonzero values in $x_{0}$ are positive. Figures 7(b), 7(c), and 7(d) show the performance improvement of TSN compared to those in Figures 6(a), 6(b), and 6(c), respectively. More details including the setting parameters and the complexity are shown in Appendix D.

V Application

V-A Nonorthogonal multiple access

We evaluated TSN using NOMA [45, 46] on an AWGN channel. We assumed that number $n$ of total users in the cell is $100$, number $s:=|\Omega|$ of active users is $10$, and number $m$ of measurements is $20$. The spreading sequence of each user is set to each column of a $20\times 100$ DFT matrix. We used the Bose-–Chaudhuri-–Hocquenghem code with codeword length $7$ and message length $4$ and applied quadrature phase shift keying modulation. Table II shows that TSN identifies the $10$ active users with probability above 90% and block error rate below $1$%, whereas the other SR algorithms have identification rate below 50% and block error rate above $10$%, regardless of SNR, i.e., E${}_{\textup{b}}$/N0. Thus, TSN reduces block error rate to less than one-tenth. Note that TSN has a complexity comparable with SBL and MMP. This shows the superiority of TSN in a real application. More details are presented in Appendix F.

V-B Image reconstruction: MNIST and OMNIGLOT

Table III and Table IV show the performance for reconstructing MNIST and OMNIGLOT [47] images of $28\times 28$ pixels in the noisy case with SNR = $25$ dB, respectively.151515We omitted to show the signal error of CoSaMP, because it has larger than 3. We randomly sampled $20$ MNIST images with sparsity equal to $215$ and OMNIGLOT images with sparsity equal to $160$. The sampled images are compressed by using real Gaussian sensing matrix $\Phi\in\mathbb{R}^{420\times 784}$. TSN recovers the MNIST and OMNIGLOT image with signal error $0.13$ and $0.17$, whereas the others have signal errors above $0.2$ and $0.4$, respectively. Thus, TSN reduces the error rate of reconstructing MNIST and OMNIGLOT images to less than half. The running time of TSN is less than one-tenth of those of SBL and MMP. Note that the DNN for TSN was trained without exploiting MNIST or OMNIGLOT image data. For the purpose of fair comparison, we excluded SR algorithms, e.g., learned denoising-based AMP (LDAMP) [6], which are required for real image data in the learning process. More details including the setting parameters and some reconstructed images are presented in Appendix G.

VI Conclusion

We propose a post-processing framework of DNN, i.e., TSN, to improve the performance of the target DNN in SR. The proposed TSN is featured by performing a tree search for support retrieval via the DNN-based index selection and extended support estimation. Experimental results demonstrate that TSN is superior to its target DNN-SR and traditional SR algorithms to uniformly recover synthesis sparse signals using diverse types of the sensing matrix, especially when the signal is difficult to be recovered, i.e., when the sampling rate $m/n$ is sufficiently small. This result implies that the performance limitation of SR can be resolved by applying the dynamic programming (tree search) to DNN.

Given that TSN solves the common inverse problem in CS and exploits a characteristic inherent in the signal distribution, we expect it to be utilized in various CS applications to enhance the signal reconstruction performance. We have verified its validity by testing two typical applications of SR.

Appendix A The training algorithm for DNN-SR used in TSN and its effect

We propose Algorithm 1 to train the DNN-SR defined by $f_{\theta}(\cdot)$ with its training parameter $\theta$. The algorithm considers noisy training data whose sparsity is in a certain range $\{k_{1}:k_{2}\}$ to improve robustness to noise and recover $x_{0}$ whose sparsity is unknown but its range is given as $\{k_{1}:k_{2}\}$. To reduce the difference between prediction error in training and test data, this method is based on the online learning method proposed in [30] such that training data is updated for each epoch.

In Algorithm 1, $n_{e}$ is the number of epochs, $s_{b}$ is the size of data batch, $s_{d}$ is the size of training data per epoch, and v_{\textup{SNR{}_{\textup{dB}}}} is SNR in decibels. We assume that sparsity $|\Omega|$ of $x_{0}$ is unknown, and its lower and upper bounds are given as $k_{1}$ and $k_{2}$, respectively. The training process uses these bounds to generate the training data. For each epoch, steps 3–7 generate training data $(x_{i},y_{i})_{i=1}^{s_{d}}$ given sensing matrix $\Phi$, where $x_{i}$ and $y_{i}$ are the synthetic signal vector and its measurement vector, respectively. Specifically, step 3 generates signal vector $x_{i}$ from the conditional probability of signal vector $x_{0}$, given its sparsity $\bar{s}$ uniformly sampled from $\{k_{1}:k_{2}\}$, which is expressed as

[TABLE]

where $p_{x_{0}}(x)$ is the distribution of $x_{0}$ and $1(\cdot)$ is the indicator function that outputs $1$ if its input statement is true, otherwise [math]. Measurement vector $y_{i}$ in step 7 corresponds to $\Phi x_{i}$ plus a vector $o_{i}$ randomly sampled such that its magnitude ranges from 0 to the expected noise magnitude $\beta$ determined by the SNR. Thus, the objective of Algorithm 1 is to train DNN-SR such that for any sparse vector $\bar{x}$ whose sparsity is in $\{k_{1}:k_{2}\}$, the network outputs the support of $\bar{x}$ from any input measurement vector $\bar{y}$ in the Euclidean ball of radius $\bar{\beta}$, centered around $\Phi\bar{x}$, such that $\left\|\bar{y}-\Phi\bar{x}\right\|\leq\bar{\beta}$, where \bar{\beta}:=\left\|\Phi\bar{x}\right\|\cdot 10^{-v_{\textup{SNR{}_{\textup{dB}}}}/20}.

To satisfy the objective, we define the following loss function in step 10, which is minimized when each signal support in training data is equal to its estimate generated by the trained DNN-SR output:

[TABLE]

where $\textup{u}_{n}(\Delta)$ with index set $\Delta\subseteq\{1:n\}$ is the function that returns an $n$-dimensional vector $v\in\mathbb{R}^{n}$, whose support is $\Delta$ and nonzero elements are equal to $1/|\Delta|$, and $\textup{ce}(a,b):=-\frac{1}{n}\sum_{i=1}^{n}b_{i}\log\,a_{i}$ is the cross entropy function for $n$-dimensional vectors $a$ and $b$. From loss function $L(\theta)$, parameter $\theta$ of the DNN-SR is updated in steps 11–12 through function $\textup{Update}(L(\theta),\eta)$, where $\eta$ is the learning rate. In Sections IV and V-A, an adaptive method for stochastic gradient descent, called RMSprop, is used for $\textup{Update}(L(\theta),\eta)$.

Optimizing DNN-SR by generating noisy training data contributes to the performance improvement of DNN-SR compared to no consideration of noisy training data. To show this, two GFLSTM networks were trained by using noiseless and noisy synthetic data with SNR equal to $5$ dB, respectively. That is, these two GFLSTM networks were trained by Algorithm 1 whose input v_{\textup{SNR{}_{\textup{dB}}}} is set to $\infty$ and $5$, respectively. Then, Table V compares the performance of both trained networks with sparsity $s$ of $x_{0}$ varied from 1 to 6 and noisy test data (SNR = $5$ dB), and then demonstrates this argument.

Appendix B TSN specification

The proposed TSN is detailed in Algorithm 5, whose objective is to recover $k$-sparse signal $x_{0}$ and its support $\Omega$ from measurement vector $y$ and sensing matrix $\Phi$. Let us define $\Omega(k)$, called the $k$-support of $x_{0}$, as any index set satisfying $\Omega(k)\supseteq\Omega$ and $|\Omega(k)|=k$.

TSN is designed by combining three algorithms, i.e., Algorithms 2, 3, and 4 presented in the sequel, and a tree search to determine $\Omega(k)$. Algorithm 2 (expand) generates multiple index sets as partial estimates of $\Omega(k)$ from an input index set. Each set generated by Algorithm 2 corresponds to a node in the tree, and hence this algorithm allows expanding the tree by generating multiple child nodes from a given parent node. Algorithm 3 (prune) reduces the number of index sets, which represent leaf nodes in the tree, to a fixed value $g$ for reducing complexity and allows to terminate TSN in an intermediate step. Algorithm 4 (initialize) is executed before the tree search to provide an initial estimate of $\Omega(k)$, which is iteratively updated through the tree search to determine the final estimate of $\Omega(k)$. The three algorithms are detailed in Section B-A and integrated into the proposed TSN in Section B-B.

B-A Algorithms composing TSN

B-A1 Algorithm 2 (expand)

Algorithm 2 takes partial support estimate $\Gamma\subseteq\{1:n\}$ of $\Omega(k)$ as one of its inputs. Step 1 generates probability vector $v:=f_{\theta}(P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y)$ from the trained DNN-SR $f_{\theta}(\cdot)$ output and residual vector $P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y$. Then, step 3 generates a remaining support estimate $Z:=\{z_{1},...,z_{u}\}\subseteq\{1:n\}\setminus\Gamma$ of size $u$ by selecting positions of the $u$-largest elements in the vector $v$. Next, Algorithm 2 generates family $\boldsymbol{S}:=(\Delta_{i})_{i=1}^{|\boldsymbol{S}|}$ of index sets (child nodes) to expand the tree by adding $z_{i}$ in $Z$ to index set $\Gamma$ (parent node).

In the noiseless case, as described in Section 1, the DNN-SR $f_{\theta}(\cdot)$ in step 1 is trained to provide the index set $Z$ in step 3 such that the following condition is satisfied for a vector $\tilde{x}\in\mathbb{K}^{n}$.

[TABLE]

Then, given that $Z$ satisfying (2) includes the true remaining support $\Omega\setminus\Gamma$ from Lemma III.1, the index in the true remaining support, which is not in the parent node, can be added to its child nodes through Algorithm 2.

B-A2 Algorithm 3 (prune)

Suppose that there exists a family $\boldsymbol{S}:=(\Delta_{i})^{|\boldsymbol{S}|}_{i=1}$ of index sets and another family $\boldsymbol{I}:=(\Lambda_{i},\bar{r}_{i})^{|\boldsymbol{I}|}_{i=1}$ of pairs, where $\Lambda_{i}$ and $\bar{r}_{i}$ in each pair in $\boldsymbol{I}$ are an index set and a signal error obtained from the set $\Lambda_{i}$, respectively. From each $\Delta_{i}$ in family $\boldsymbol{S}$, Algorithm 3 determines the $k$-support estimate and its signal error. Then, the algorithm reduces the number of index sets in $\boldsymbol{S}$ and $\boldsymbol{I}$ by selecting a family $\boldsymbol{D}$ of $g$ index sets with the smallest signal errors among their union $\{\Pi_{1}:=\Delta_{1},...,\Pi_{|\boldsymbol{S}|}:=\Delta_{|\boldsymbol{S}|},\Pi_{|\boldsymbol{S}|+1}:=\Lambda_{1},...,\Pi_{|\boldsymbol{S}|+|\boldsymbol{I}|}:=\Lambda_{|\boldsymbol{I}|}\}$. Given that the index sets in $\boldsymbol{S}$ and $\boldsymbol{I}$ represent leaf nodes in the search tree, the algorithm prunes the tree to reduce the number of nodes.

To obtain the signal error given partial support estimate $\Pi_{t}$, extended support estimate $\Psi_{t}$ of size $m-1$ for $t\in\{1:|\boldsymbol{S}|\}$ is obtained by adding remaining support estimate $Z$ of $u:=m-1-|\Pi_{t}|$ indices to index set $\Pi_{t}$ in steps 2–4. Set $Z\subseteq\{1:n\}\setminus\Pi_{t}$ in step 3 is obtained through the DNN-based index selection by selecting the position of the $u$-largest elements in probability vector $v:=f_{\theta}(P^{\perp}_{\mathcal{R}(\Phi_{\Pi_{t}})}y)$ generated from the trained DNN-SR and residual vector $P^{\perp}_{\mathcal{R}(\Phi_{\Pi_{t}})}y$.161616Similar to step 3 of Algorithm 2, set $Z$ in step 3 of Algorithm 3 includes the remaining support, $\Omega\setminus\Pi_{t}$, if DNN-SR defined by $f_{\theta}(\cdot)$ is ideally trained, as explained in Sections 1 and B-A1. Steps 5–6 generate estimate $\bar{\Omega}_{t}$ of $\Omega(k)$ by selecting $k$ indices from extended support estimate $\Psi_{t}$. Specifically, step 5 corresponds to a ridge regression with parameters $(\gamma,\eta)$, which are obtained by minimizing the following cost

[TABLE]

where $\bar{\Phi}=\Phi_{\Psi_{t}}\in\mathbb{K}^{m\times a}$ is a submatrix of $\Phi$, whose columns are indexed by $\Psi_{t}$, $\Sigma=(\bar{\eta}^{-2}{\bar{\Phi}}^{*}{\bar{\Phi}}+D(\bar{\gamma}))^{-1}$, and $D(\bar{\gamma})$ is the diagonal matrix whose $i$-th diagonal element is $\gamma_{i}$ in $\bar{\gamma}=(\gamma_{1},...,\gamma_{a})^{\top}$. The ridge regression is derived through the Bayesian methodology [31] for solving linear inverse problems to determine vector $\bar{x}\in\mathbb{K}^{n}$ such that (4) holds.

[TABLE]

Furthermore, in the noisy case, the ridge regression considers the noise magnitude corresponding to $\eta$ to improve noise robustness compared to the least-squares regression. Next, we obtain an approximate solution of $(\ref{sble})$ using SBL to reduce complexity. In the numerical experiments, we set the regularization parameter and maximum iteration number of the SBL used in TSN to \max[(\left\|y\right\|\cdot 10^{-v_{\textup{SNR{}_{\textup{dB}}}}/20})^{2}/m,10^{-4}] and $10$, respectively. In the noiseless case, parameters $(\gamma,\eta)$ are set to zero, and hence the ridge regression is equivalent to the least-squares regression, i.e., $(\Phi^{*}_{{\Psi_{t}}}\Phi_{{\Psi_{t}}})^{-1}\Phi^{*}_{{\Psi_{t}}}y=\underset{x}{\arg\min}\left\|\Phi_{\Psi_{t}}x-y\right\|$. The least-squares method has a lower complexity than SBL and provides $x_{0}$ as the unique solution of (4) in the noiseless case ($w=0$) if $\Psi_{t}\supseteq\Omega$, $|\Psi_{t}|\leq m$, and $\Phi_{\Psi_{t}}$ has full column rank. Note that the probability of satisfying $\Psi_{t}\supseteq\Omega$ increases with the dimension of $\Psi_{t}$. In addition, if $\Psi_{t}\supseteq\Omega$ holds, the inversion problem of SR can be simplified as a problem where $\Phi$ is replaced by submatrix $\Phi_{\Psi_{t}}$. However, we also note that the signal error $\underset{x}{\min}\left\|\Phi_{\Psi_{t}}x-y\right\|$ estimated by using any $\Psi_{t}$ of size $m$ is equal to zero, even if $\Psi_{t}\nsupseteq\Omega$. To avoid this trivial case and maximize the probability of satisfying $\Psi_{t}\supseteq\Omega$, we set the dimension of extended support estimate $\Psi_{t}$ to $m-1$.171717The extended support is defined by an index set including $\Omega$. Some popular existing SR algorithms gOMP [37], SP [39], and CoSaMP [48] exploit an extended support estimate in the process to estimate $\Omega$. However, these algorithms do not utilize a DNN-based index selection. Besides, the size of the extended support estimate $\Psi_{t}$ in TSN is set to $m-1$, which is indepedent of $s$, to maximize the probability satisfying $\Psi_{t}\supseteq\Omega$, whereas its size in gOMP, SP, and CoSaMP is $t\cdot\min(s,\left\lfloor{m/t}\right\rfloor)$ , $2s$, and $3s$, respectively, for a constant $t$ smaller than $s$, where $s:=|\Omega|$ is the sparsity of $x_{0}$.

Then, we calculate signal error $r_{t}:=\underset{x}{\min}\left\|\Phi_{{\bar{\Omega}_{t}}}x-y\right\|$ using $k$-support estimate $\bar{\Omega}_{t}$, which is obtained in step 7. If there exists a norm $r_{i}$ for $i\in\{1:|\boldsymbol{S}|\}$ smaller than a threshold $\epsilon$, $\bar{\Omega}_{i}$ is considered as the true $k$-support and Algorithm 3 terminates by returning $\bar{\Omega}_{i}$ and setting Boolean parameter $e$ to true, indicating the successful termination of TSN. Otherwise, if the number $g$ of index sets remained after pruning is larger than 1, Algorithm 3 goes to step 14 to select, from family $(\Pi)_{i=1}^{|\boldsymbol{S}|+|\boldsymbol{I}|}$ of index sets, set $\Theta$ of $g$ indices $i$ minimizing signal error $r_{i}$. Then, Algorithm 3 returns set $\boldsymbol{D}$ composed of pairs $(\Pi_{t},r_{t})_{t\in\Theta}$ in step 15; Algorithm 3 only leaves $g$ index sets $(\Pi)_{i\in\Theta}$ from the $|\boldsymbol{S}|+|\boldsymbol{I}|$ index sets $(\Pi)_{i=1}^{|\boldsymbol{S}|+|\boldsymbol{I}|}$. Note that in step 16 a $k$-support estimate $\bar{\Omega}_{q}$, where $q$ is the argument $i$ with the minimum residual error $r_{i}$ for $i\in\Theta$, is selected among $g$ index sets $(\Pi)_{i\in\Theta}$. In addition, Algorithm 3 takes pair $(\check{\Omega},\check{r})$ as its input, where $\check{\Omega}$ is an index set for the $k$-support estimate and $\check{r}$ is its signal error. Then, the algorithm replaces the input pair $(\check{\Omega},\check{r})$ by $(\bar{\Omega}_{q},r_{q})$ in step 18, provided that signal error $r_{q}$ generated from $k$-support estimate $\bar{\Omega}_{q}$ is smaller than $\check{r}$. In the case where only one index set is remained after pruning, i.e., $g=1$, Algorithm 3 generates set $\Theta$ of $z$ indices $i$ minimizing signal error $r_{i}$ in step 21 and returns index set $J:=\underset{i\in\Theta}{\cup}\Pi_{i}$ by unioning $z$ sets $(\Pi_{i})_{i\in\Theta}$ indexed by $i\in\Theta$ in step 22; the input value $z$ of TSN represents the number of sets to be combined as one set. Similarly with step 18, Algorithm 3 updates pair $(\check{\Omega},\check{r})$ by obtaining a $k$-support estimate $\dot{\Omega}$ and its signal error $\dot{r}$ from index set $J$ in steps 22 and 25.

B-A3 Algorithm 4 (Initialize)

Algorithm 4 provides index set $\check{\Omega}$, an initial estimate for $\Omega(k)$ in TSN (Algorithm 5), given the tuple ($\Phi,y,k$) in the following three stages.

The first stage (steps 1-2) aim to obtain two $k$-support estimates, $\Delta_{\textup{DNN}}$ and $\Delta_{\textup{OMP}}$, from the DNN-based and OMP-based selections, respectively.181818In step 1, $k$-support estimate $\Delta_{\textup{DNN}}$ is obtained by selecting positions of the $k$-largest elements in probability vector $v:=f_{\theta}(y)$, which is generated from the trained DNN-SR $f_{\theta}(\cdot)$ output and the measurement vector $y$. In step 2, another $k$-support estimate $\Delta_{\textup{OMP}}$ is obtained by selecting $k$ indices via the OMP-based rule. The second stage (steps 3–7) generates extended support estimate $\Psi$ of size $m-1$ from two $k$-support estimates. For this, the signal errors, $r_{\textup{DNN}}$ and $r_{\textup{OMP}}$, are obtained through estimates $\Delta_{\textup{DNN}}$ and $\Delta_{\textup{OMP}}$ to select one estimate among them. If $r_{\textup{DNN}}$ is smaller than $r_{\textup{OMP}}$, $\Delta_{\textup{DNN}}$ includes more elements in the true support $\Omega$ than $\Delta_{\textup{OMP}}$. Thus, extended support estimate $\Psi=T_{m-1}(f_{\theta}(y))$ is obtained via the DNN-based index selection in step $4$ when $r_{\textup{DNN}}$ is smaller than $r_{\textup{OMP}}$. Otherwise, OMP-based index selection is used to generate set $\Psi$ as $\Gamma_{m-1}$ in step 6. The third stage (steps 8–13) generates theinitial estimate $\check{\Omega}$ for $\Omega(k)$ in TSN via the same approach from steps $5$–$6$ of Algorithm 3 using extended support estimate $\Psi$.

B-B TSN procedure

The proposed TSN (Algorithm 5) obtains $(\hat{x},\hat{\Omega})$ as an estimate of $x_{0}$ and its support $\Omega$ given input tuple ($\Phi,y,k$) and a DNN-SR $f_{\theta}(\cdot)$ trained by Algorithm 1.

First, TSN runs Algorithm 4 in step 1 to obtain $k$-support estimate $\check{\Omega}$ and determine whether its signal error is below threshold $\epsilon$. If the error is over the threshold, i.e., $e=0$, TSN goes to steps 2–22 to update $k$-support estimate $\check{\Omega}$ through the tree search. Then, steps 23–26 in TSN return $(\hat{x},\hat{\Omega})$ from $k$-support estimate $\check{\Omega}$ obtained in steps 1–22.

In step 15, $\boldsymbol{G}_{a}$ is a family of index sets generated from $\boldsymbol{G}_{a-1}$ by executing the loop in steps 4–21, where $a$ is a positive integer with initial value $1$ and incremented by $1$ at every iteration, and $\boldsymbol{G}_{0}$ is the empty set. The procedure to generate $\boldsymbol{G}_{a}$ from $\boldsymbol{G}_{a-1}$ is explained in the sequel and illustrated in Figure 8, which mainly consists of expanding and pruning the search tree.

(Steps 7–14: expand) Suppose that family $\boldsymbol{G}_{a-1}:=(\Delta_{i})_{i=1}^{|\boldsymbol{G}_{a-1}|}$ of index sets is available, where each set $\Delta_{i}$ represents a support estimate of size $\sum_{j=1}^{a-1}l_{j}$ for $i\in\{1:|\boldsymbol{G}_{a-1}|\}$. Algorithm 2 is executed by taking each index set $\Delta_{i}$ as its input and returns a family $\boldsymbol{S}^{i}_{1}:=(\Lambda_{j})_{j=1}^{|\boldsymbol{S}^{i}_{1}|}$ of index sets in step 7, where each set $\Lambda_{l}$ for $l\in\{1:|\boldsymbol{S}^{i}_{1}|\}$ is a support estimate of size $1+|\Delta_{i}|$, which is an extension of $\Delta_{i}$ by adding one index. Hence, multiple child nodes $(\Lambda_{l})_{l\in\{1:|\boldsymbol{S}^{i}_{1}|\}}$ are generated from each parent node $\Delta_{i}$ to expand the search tree. By repeating the same procedure $l_{a}-1$ times in steps 9–14, TSN expands the tree such that family $\boldsymbol{S}^{i}_{l_{a}}$ of sets having $\sum_{i=1}^{a}l_{i}$ indices is obtained from each index set $\Delta_{i}$ in $\boldsymbol{G}_{a-1}$.

(Step 15: prune) Once each family $\boldsymbol{S}^{i}_{l_{a}}$ of index sets is obtained for $i\in\{1:|\boldsymbol{G}_{a-1}|\}$ after step 14, Algorithm 3 is executed in step 15 to update $\boldsymbol{G}_{a}$ by selecting $g_{a}$ index sets with smallest signal errors among index sets in $\boldsymbol{S}^{i}_{l_{a}}$ and $\boldsymbol{G}_{a}$. Note that Algorithm 3 (steps 6–7 in Algorithm 3) in step 15 generates pairs of a temporary $k$-support estimate and its signal error from each index set in $\boldsymbol{S}^{i}_{l_{a}}$ and selects one $k$-support estimate $\dot{\Omega}$ with the smallest signal error among the pairs if $g_{a}$ is larger than 1.191919$k$-support estimate $\dot{\Omega}$ is obtained in step 20 in Algorithm 3 if $g_{a}$ equal to 1. That is, set $\dot{\Omega}$ represents a $k$-support estimate obtained from the nodes existing at the tree depth $a$. Then, Algorithm 3 takes $\check{\Omega}$, which is the $k$-support estimate generated from nodes at the tree depth smaller than $a$, as its input and updates $\check{\Omega}$ as $\bar{\Omega}$ if the signal error $\check{r}$ corresponding to $\check{\Omega}$ is larger than that obtained from $\bar{\Omega}$. If the signal error generated from $k$-support estimate $\check{\Omega}$ is below threshold $\epsilon$ ($e=0$), TSN terminates the tree search and goes to step 23 for estimating $x_{0}$ and $\Omega$ from $\check{\Omega}$; set $\check{\Omega}$ in step 23 indicates the final $k$-support estimate via the tree search in TSN.

Steps $23$–$25$ determine the final estimate $\hat{\Omega}$ of true support $\Omega$ from $k$-support estimate $\check{\Omega}$ by using the following two-stage process. In the first stage, signal estimate $\bar{x}:=(\bar{x}_{1},...,\bar{x}_{n})$, whose nonzero elements are supported on $\check{\Omega}$, is generated through the ridge regression shown in (3). Then, in the second stage, support $\Omega$ is estimated as $\hat{\Omega}$ by selecting indices $i$ of $\bar{x}_{i}$ whose absolute values $\operatorname{abs}(\bar{x}_{i})$ are larger than a threshold $\rho$. We set this threshold to $v/2$, where $v$ is the minimum among the absolute values of a possible signal vector.202020If the sparsity $|\Omega|$ is given to TSN, steps $23$–$24$ in TSN can be omitted by setting the TSN input $k$ to $|\Omega|$ and $\hat{\Omega}$ to $\check{\Omega}$. Then, estimated signal vector $\hat{x}$ is generated in step 25 by the least-squares method from support estimate $\hat{\Omega}$, and provided along with $\hat{\Omega}$ as the final TSN output.

Appendix C Proof of Lemma III.1

From the assumptions that $|\Omega|<m$ and every $m$ columns in $\Phi$ exhibit full rank, the condition $P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y\in\mathcal{R}(\Phi_{\Omega\setminus\Gamma})$ holds given any index set $\Gamma\subseteq\{1:n\}$. Suppose that $P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y$ does not belong to the following subspace $E$ ($P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y\notin E$)

[TABLE]

Then, $P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y$ belongs to $\mathcal{R}(\Phi_{\Omega\setminus\Gamma})\setminus E$. Given that (2) implies that $P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y\in\mathcal{R}(\Phi_{D})$, $P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y\in\mathcal{R}(\Phi_{D})\cap(\mathcal{R}(\Phi_{\Omega\setminus\Gamma})\setminus E)$ holds for any index set $D$ satisfying (2). Note that the condition $P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y\in\mathcal{R}(\Phi_{D})\cap(\mathcal{R}(\Phi_{\Omega\setminus\Gamma})\setminus E)$ implies that ${D}\supseteq\Omega\setminus\Gamma$. Thus, ${D}\supseteq\Omega\setminus\Gamma$ holds for any index set $D$ satisfying (2) if $P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y\notin E$. Given that from the assumption for $\Phi$, the rank of $\mathcal{R}(\Phi_{\Omega\setminus\Gamma})$ is strictly larger than that of $\mathcal{R}(\Phi_{J})\cap\mathcal{R}(\Phi_{\Omega\setminus\Gamma})$ for any index set $J\subseteq\{1:n\}$ such that $|J\cap(\Omega\setminus\Gamma)|<|\Omega\setminus\Gamma|$ and $|J|<m$, the event region satisfying $P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y\in E$ has Lebesgue measure zero on the range space $\mathcal{R}(\Phi_{\Omega\setminus\Gamma})$ so that the condition $P^{\perp}_{\mathcal{R}(\Phi_{\Gamma})}y\notin E$ is satisfied almost surely.

Appendix D Performance comparison given complex-valued measurements

In this section, the performance of TSN is compared to other existing SR algorithms shown in Section IV-A given complex-valued measurements. Let $\mathcal{CN}(a,b)$ denote the complex Gaussian distribution whose real and imaginary parts have mean $a$ and variance $b$, respectively. Real and imaginary parts of each nonzero elements in $x_{0}$ are independently and uniformly sampled from $-1$ to $1$, excluding the interval from $-$0.1 to 0.1, respectively. The measurement noise $w$ follows $\mathcal{CN}(0,\sigma^{2}_{w})$ with $\sigma_{w}$ dependent on the given SNR. Algorithm 1 whose input $(k_{1},k_{2})$ is set to $(1,15)$ is used to train the GFLSTM-based network $f_{\theta}(\cdot)$. The sparsity $s$ of $x_{0}$ is varied from [math] to $12$, which is given to SR algorithms except for TSN. The input $k$ of TSN is fixed to $12$. The other parameters for simulation setting are set equal to those in Section IV-A1.

To demonstrate the superiority of TSN over other algorithms for the case of using various types of the sensing matrix $\Phi$, we have set $\Phi$ in one of the following three matrices: a complex Gaussian matrix, a partial DFT matrix, and a complex matrix with highly correlated columns [30, 43]. These matrices are generated according to the following rules and the columns of $\Phi$ are $l_{2}$-normalized.

• •

(Complex Gaussian matrix) Each element of $\Phi\in\mathbb{C}^{m\times n}$ follows $\mathcal{CN}(0,1)$.

• •

(Partial DFT matrix) $m$ rows are randomly selected from the DFT matrix of size $n\times n$.

• •

(Matrix with correlated columns) $\Phi=\sum_{z=1}^{n}[\frac{1}{z^{2}}p^{1}_{z}(q^{1}_{z})^{\top}+\sqrt{-1}\cdot(\frac{1}{z^{2}}p^{2}_{z}(q^{2}_{z})^{\top})]$ where $\sqrt{-1}$ is the imaginary unit and each element in $p^{i}_{z}\in\mathbb{R}^{m}$ ($q^{i}_{z}\in\mathbb{R}^{n}$) is drawn independently from a standard Gaussian distribution $\mathcal{N}(0,1)$ for $i\in\{1:2\}$ and $z\in\{1:n\}$.

Figures 9, 10, and 11 show the performance comparison of each algorithm when $\Phi$ is set to the complex Gaussian matrix, the partial DFT matrix, and the complex matrix with correlated columns, respectively. It is observed from the results that the performance of SR algorithms shows the same trend as that in Figure 5 in Section IV; TSN shows better performance than existing SR algorithms in both noiseless and noisy cases. This indicates that TSN can verify to be applied to both real or complex signal restoration. In particular, it is observed in Figures 9(a), 10(a), and 11(a) that TSN almost achieves the ideal limit of sparsity $(s=10)$ for the uniform recovery of $x_{0}$ with execution time of about 1 second, regardless of the sensing matrix type.

Furthermore, TSN can be applied to recover structured sparse signals and show better performance than existing SR algorithms. To provide an example, we consider the case when a non-negative constraint is added to the signal distributions used to plot Figures 5, 9, 10, and 11 such that the real and imaginary parts of nonzero elements in $x_{0}$ are uniformly and independently generated from $0.1$ to $1$. The sparsity $s$ of $x_{0}$ is varied from [math] to $15$ and given to SR algorithms except for TSN whose input $k$ is set to $10$ in the real-valued case or $15$ in the complex-valued case. The other setting parameters are the same as those used to plot the figures. Figure 12 shows the performance evaluation of each algorithm in this experimental environment. It is observed from Figure 12 that TSN enhances the performance of its target DNN-SR, i.e., GFLSTM, and significantly outperforms existing SR algorithms given real-valued or complex-valued measurements. In particular, it is observed that TSN achieves the ideal limit of sparsity $(s=10)$ for the uniform recovery of $x_{0}$ with execution time smaller than $0.1$ second in the complex-valued case. It is also observed in the real-valued case (Figures 12(a)-(c)) that the signal recovery rate of TSN is larger than 90 percent with running time about 1 second when the sparsity $s$ is set to its ideal limit $10$. Figures 12(b) and 12(e) show that the maximum sparsity $s$ for the uniform recovery of $x_{0}$ using TSN is larger than two times that using SBL when $\Phi$ is set to real-valued or complex-valued Gaussian matrix.

Appendix E Scalability

We set the hidden unit size of GFLSTM to 2000, number $N_{\textup{max}}$ of path candidates in MMP to 20000, and input parameters $(q,\boldsymbol{l},\boldsymbol{g},t_{\textup{max}})$ of TSN, i.e., Algorithm 5, to (m, (2,1,2,1,2,1,2,1), (60,1,60,1,60,1,60,1), 10). We used Algorithm 1 whose input $(k_{1},k_{2})$ is set to $(1,m/2)$ to train GFLSTM and used the trained GFLSTM as the DNN $f_{\theta}(\cdot)$ in TSN. For other parameters, we used the same settings shown in Figure 5(a). Then, we selected the four highest performing algorithms, i.e., TSN, MMP, SBL, and Lasso shown in Table I and performed an additional evaluation test for $m>40$ and $n=10^{4}$.212121We only tested until $(m,n)\leq(1500,10^{4})$ due to insufficient memory for MMP. The results under these settings are listed in Table VI, where TSN outperforms the other algorithms and has a lower complexity than MMP and SBL. Hence, TSN scales well at least for the case where $m$ is less than 1500. However, as $m$ increases, we observed that the ratio of performance versus complexity of TSN tends to decrease relative to other SR algorithms.

Appendix F Application: NOMA

NOMA has been widely adopted into upcoming 5G wireless networks to enhance spectral efficiency. We propose Algorithm 6 to detail NOMA. We set number $n$ of users in the cell and number $m$ of measurements to $100$ and $20$, respectively. The spreading sequence of each user corresponds to each column of $20\times 100$ partial DFT matrix $\Phi$. In addition, we set number $s$ of active users in the cell to $10$. Support set $\Omega$ of size $s$ was uniformly sampled from $\{1:n\}$, with each element representing an index for each of the $s$ active users. We randomly generated set $z_{m}$ of $s$ message blocks, each consisting of $4$ bits and assigned to each active user. We used Bose-–Chaudhuri-–Hocquenghem code with codeword length $7$ and message length $4$ for the encoding and decoding steps 2 and 12 at coding rate $r_{c}$ of $4/7$. Then, we applied quadrature phase shift keying for the modulation and demodulation steps 3 and 11, retrieving $r_{m}=2$ bits per symbol. In step 4, we used the AWGN channel by sampling each element of noise vector $w_{i}$ from a Gaussian distribution with zero mean and variance determined by the target SNR.

The goal of NOMA is to recover user index set $\Omega$ and set $z_{m}$ of message blocks for the $s$ active users. In step 5, we run a target SR algorithm to obtain support candidate $\bar{\Omega}_{i}$ from each measurement vector $y_{i}$. Then, the final estimate $\hat{\Omega}$ for support $\Omega$ is obtained in step 9 by selecting the $s$ most frequent indices in candidates $(\bar{\Omega}_{i})_{i=1}^{\lceil B/(r_{m}\cdot r_{c})\rceil}$. From estimate $\hat{\Omega}$, signal matrix $X$ is estimated as $\hat{X}$ using least-squares regression in step 10. Finally, set $z_{m}$ of $s$ message blocks is reconstructed as $\hat{z}_{m}$ during demodulation and decoding steps 11 and 12 by using estimate $\hat{X}$. To evaluate the proposed TSN, we used TSN3 to plot Figure 10. The DNN, i.e., GFLSTM, used in TSN was trained as described in Section D and the other setting parameters are the same as those used to plot Figure 10.

Appendix G Application: image reconstruction

We adjusted the size of OMNIGLOT to $28\times 28$ pixels. We have observed that 95% of the MNIST or the resized OMNIGLOT images have sparsity below 215 and 160, respectively. Consequently, we randomly sampled the MNIST images with sparsity 215 and the OMNIGLOT images with sparsity 160, and evaluated the performance of SR algorithms. Figures 13–15 and Figures 16–18 show some MNIST and OMNIGLOT images reconstructed by each SR algorithm, respectively.222222We evaluated the performance of TSN against the existing SR algorithms shown in Section IV except for MMP2, because of its long execution time; MMP shown in Table III and Figures 13–15 indicates MMP1 shown in Section IV. Images labeled with ‘difference’ indicate the difference between the output image generated from the corresponding SR algorithm and the original image. Similar performance was observed for other MNIST or OMNIGLOT images.

We used the GFLSTM network for DNN $f_{\theta}(\cdot)$ in TSN and trained the network using Algorithm 1, whose input tuple (k_{1},k_{2},s_{d},s_{b},n_{e},v_{\textup{SNR{}_{\textup{dB}}}}) was set to $(1,250,6\cdot 10^{5},250,400,20)$. Note that DNN $f_{\theta}(\cdot)$ in TSN was trained without using MNIST images. Therefore, we assumed that each signal vector used as training data in Algorithm 1 was generated such that each of its nonzero elements is independently and uniformly sampled from [math] to $1$.232323During testing, we normalized the MNIST image by dividing its elements by 256 to obtain values between 0 and 1. We evaluated the proposed TSN (Algorithm 5) using the following set, $\boldsymbol{\tau}$, of input parameters:

[TABLE]

where \bar{\epsilon}:=\max[(\left\|y\right\|\cdot 10^{-\bar{v}_{\textup{SNR{}{\textup{dB}}}}/20}),10^{-5}] is the signal error bound and \bar{v}_{\textup{SNR{}{\textup{dB}}}}=25 is the SNR in decibels.