How to exploit prior information in low-complexity models

TL;DR

This paper develops a method to optimally exploit prior information in low-complexity models for compressed sensing, improving signal recovery with fewer measurements by choosing weights based on probabilistic structure.

Contribution

It introduces a generalized non-uniform model with probabilistic partitions and provides a method to select unique optimal weights using conic integral geometry, applicable to various structured signals.

Findings

Optimal weights reduce the number of measurements needed

Weighted convex optimization outperforms unweighted methods

Method applies to diverse low-dimensional models like sparsity and total variation

Abstract

Compressed Sensing refers to extracting a low-dimensional structured signal of interest from its incomplete random linear observations. A line of recent work has studied that, with the extra prior information about the signal, one can recover the signal with much fewer observations. For this purpose, the general approach is to solve weighted convex function minimization problem. In such settings, the convex function is chosen to promote the low-dimensional structure and the optimal weights are so chosen to reduce the number of measurements required for the optimization problem. In this paper, we consider a generalized non-uniform model in which the structured signal falls into some partitions, with entries of each partition having a definite probability to be an element of the structure support. Given these probabilities and regarding the recent developments in conic integral geometry, we provide a method to choose the unique optimal weights for any general low-dimensional signal model. This class of low-dimensional signal model includes many popular examples such as $\ell_1$ analysis (entry-wise sparsity in an arbitrary redundant dictionary), $\ell_{1,2}$ norm (block sparsity) and total variation semi-norm (for piece-wise constant signals). We show through precise analysis and simulations that the weighted convex optimization problem significantly improves the regular convex optimization problem as we choose the unique optimal weights.

Equations

$${\mathsf{P}}_0:$$

$$s.t.\mathbf{Az}=\mathbf{b},$$

$${\mathsf{P}}_1:$$

$$s.t.\mathbf{Az}=\mathbf{b}.$$

$${\mathsf{P}}_{1,\mathbf{w}}:$$

$$s.t.\mathbf{Az}=\mathbf{b},$$

$${\mathsf{P}}_{1,\mathbf{\Omega },\mathbf{w}}:$$

$$s.t.\mathbf{Az}=\mathbf{b},$$

$${\mathsf{P}}_{1,2,\mathbf{w}}:$$

$$s.t.\mathbf{Az}=\mathbf{b},$$

$${\mathsf{P}}_{\mathrm{TV},\mathbf{w}}:$$

$$s.t.\mathbf{Az}=\mathbf{b},$$

$$\partial f(\mathbf{x}):=\{\mathbf{z}\in {\mathbb{R}}^n:f(\mathbf{y})\ge f(\mathbf{x})+⟨\mathbf{z},\mathbf{y}-\mathbf{x}⟩:\forall \mathbf{y}\in {\mathbb{R}}^n\}.$$

$$\partial f(\mathbf{x})=\{\mathbf{z}\in {\mathbb{R}}^n:⟨\mathbf{z},\mathbf{x}⟩=f(\mathbf{x}),f^{\ast }(\mathbf{z})=1\},$$

$$f^{\ast }(\mathbf{z})=\underset{f(\mathbf{v})=1}{\sup }⟨\mathbf{z},\mathbf{v}⟩\le \underset{f(\mathbf{v})=1}{\sup }(f(\mathbf{v}+\mathbf{x})-f(\mathbf{x}))\le 1,$$

$$f(\mathbf{x})+⟨\mathbf{z},\mathbf{y}-\mathbf{x}⟩=⟨\mathbf{z},\mathbf{x}⟩+⟨\mathbf{z},\mathbf{y}-\mathbf{x}⟩\le f(\mathbf{y})\cdot$$

$$\mathcal{D}(f,\mathbf{x})=\bigcup _{t\ge 0}\{\mathbf{z}\in {\mathbb{R}}^n:f(\mathbf{x}+t\mathbf{z})\le f(\mathbf{x})\}\cdot$$

$${\mathcal{N}}_f(\mathbf{x})={\mathcal{D}}^{\circ }(f,\mathbf{x})=\{\mathbf{y}\in {\mathbb{R}}^n:⟨\mathbf{y},\mathbf{z}⟩\le 0:\forall \mathbf{z}\in \mathcal{D}(f,\mathbf{x})\}.$$

$${\mathcal{D}}^{\circ }(f,\mathbf{x})=\mathrm{cone}(\partial f(\mathbf{x})):=\bigcup _{t\ge 0}t.\partial f(\mathbf{x}).$$

$$\omega (\mathcal{C}):=\mathbb{E}\underset{\mathbf{v}\in \mathcal{C}}{\sup }⟨\mathbf{g},\mathbf{v}⟩,$$

$$\delta (\mathcal{C}):=\text{mathds}E\Vert {\mathcal{P}}_{\mathcal{C}}(g){\Vert }_2^2=\text{mathds}E{\mathrm{dist}}^2(g,{\mathcal{C}}^{\circ }),$$

$${\mathsf{P}}_f:\underset{\mathbf{x}\in {\mathbb{R}}^n}{\min }f(\mathbf{x})s.t.\mathbf{Ax}=\mathbf{b}.$$

$$0\le \underset{t\ge 0}{\inf }\text{mathds}E{\mathrm{dist}}^2(\mathbf{g},t\partial f(\mathbf{x}))-\delta (\mathcal{D}(f,\mathbf{x}))\le \frac{2{\sup }_{s\in \partial f(\mathbf{x})}\Vert s{\Vert }_2}{f(\frac{\mathbf{x}}{\Vert \mathbf{x}{\Vert }_2})}.$$

$$m\ge {\kappa }^2(\mathbf{\Omega })\delta (\mathcal{D}(\Vert .{\Vert }_1,\mathbf{\Omega }\mathbf{x}))+1+\sqrt{8\log \frac{4}{\eta }n},$$

$$\displaystyle\mathds{P}(\mathcal{D}(\|\bm{\Omega}.\|_{1},\bm{x})\cap\mathrm{null}(\bm{A})=\{\bm{0}\})\geq 1-\eta\exp\big{(}-\frac{\mu^{2}}{8n}\big{)},$$

$$m_{p,s}$$

$$m_{p,s,\mathbf{w}}$$

$$\delta (\mathcal{D}(\Vert \mathbf{\Omega }.{\Vert }_1,\mathbf{x}))\le {\kappa }^2(\mathbf{\Omega })\delta (\mathcal{D}(\Vert \cdot {\Vert }_1,\mathbf{\Omega }\mathbf{x}))+1.$$

$${\hat{m}}_{p,s}:={\kappa }^2(\mathbf{\Omega })\underset{t\ge 0}{\inf }{\Psi }_t(\sigma )+\frac{1}{p},$$

$${\Psi }_t(\sigma )=\sigma (1+t^2)+(1-\sigma )\varphi (t),$$

$${\hat{m}}_{p,s,\mathbf{w}}:={\kappa }^2(\mathbf{\Omega })\underset{t\ge 0}{\inf }{\Psi }_{t,\mathbf{w}}(\sigma ,\mathbf{\rho },\mathbf{\alpha })+\frac{1}{p},$$

$$\displaystyle\Psi_{t,\bm{w}}(\sigma,\bm{\rho},\bm{\alpha})=\sum_{i=1}^{L}\rho_{i}\big{(}\alpha_{i}(t^{2}\omega_{i}^{2}+1)+(1-\alpha_{i})\phi(t\omega_{i})\big{)}.$$

$$p^{-1}\delta (\mathcal{D}(\Vert .{\Vert }_{1,\mathbf{w}},\mathbf{\Omega }\mathbf{x}))\le \underset{t\ge 0}{\inf }\underset{{\Psi }_{t,\mathbf{w}}(\sigma ,\rho ,\alpha )}{\underbrace{p^{-1}\text{mathds}E{\mathrm{dist}}^2(\mathbf{g},t\partial \Vert .{\Vert }_{1,\mathbf{w}}(\mathbf{\Omega }\mathbf{x}))}}.$$

Full text

I Introduction

Over the past decade, it has become an established fact that based on prior information about the sparsity feature of the signal111Signals with this feature have much less nonzero elements than the ambient dimension. $\bm{x}\in\mathbb{R}^{n}$, one can recover it from a few random linear measurements known as compressed sensing (CS)[1]. For this purpose, it is necessary to introduce a function that promotes sparsity. This leads to the following optimization problem (known as $\ell_{0}$ problem or $\mathsf{P}_{0}$):

[TABLE]

where in (I), $\bm{A}\in\mathbb{R}^{m\times n}$ and $\bm{b}\in\mathbb{R}^{m}$ are the measurement matrix and measurement vector, respectively222We typically consider the case that $s$ is much smaller than $n$. . The reason behind this optimization problem is that among all possible solutions of the equation $\bm{Az}=\bm{b}$, the sparsest one is unique [1].

The optimization problem $\mathsf{P}_{0}$ is non-convex, computationally intractable, and in fact NP-hard. Although there has not ever been any algorithm that can solve $\mathsf{P}_{0}$ in polynomial time, but if it existed, it would require $\mathcal{O}(s)$ measurements in the case that the original signal $\bm{x}\in\mathbb{R}^{n}$ is $s$-sparse333$\|\bm{x}\|_{0}\leq s$.

Candes and Tao in [2] and Donoho in [1] proved that under the so-called Restricted Isometry Property (RIP) on the measurement matrix $\bm{A}\in\mathbb{R}^{m\times N}$ , instead of $\mathsf{P}_{0}$, one can alternatively solve the following optimization problem known as $\ell_{1}$ minimization problem or $\mathsf{P}_{1}$ and recover the original $s$-sparse signal from $\mathcal{O}(s\log(\frac{n}{s}))$ random linear measurements $\bm{y}\in\mathbb{R}^{m}$. By $\mathsf{P}_{1}$, we pay for a tractable solution to $\bm{Az}=\bm{b}$ with suboptimal number of measurements.

[TABLE]

Ever since, most of the literature on compressed sensing has focused on the case where the only side information is that the signal of interest is sparse. In many applications however, there may exist some other extra information about the underlying signal in addition to sparsity. It’s plausible that we can do even better than the case that the only information is sparsity. The common way to exploit both sparsity and extra information without entering the non-convex optimization world is weighted convex minimization defined as follows:

[TABLE]

where in (I), $\bm{w}\in\mathbb{R}_{+}^{n}$ refers to the positive weights applied to the components of the structured signal. Note that, a strictly structured sparse signal is characterized by two parameters:(1) the set of non-zero locations (support) and (2) the signal values. If one has the signal support, the undetermined equation $\bm{Az}=\bm{b}$ reduces to the solvable overdetermined problem. However, if one has an estimated support which is only fractionally wrong it’s better to associate larger weights to zero entries and smaller weights to the non-zeros. This intuition was first analysed by Candes et. al. in [3]. A major question that has been the subject of recent researches ([4, 5, 6, 7, 8]) is how to suitably choose the weights in $\mathsf{P}_{1,\bm{w}}$. In other words, given prior information, how we can compute optimal weights. We mean optimal444Although this meaning is not universal in the CS literature. weights in the sense that the required number of measurements for $\mathsf{P}_{1,\bm{w}}$ to succeed with overwhelming probability, is minimized.

I-A Related Works

Von Borries et al. in [9], showed that the number of measurements needed to recover a sparse signal in discrete fourier transform (DFT) by weighted-$\ell_{1}$ analysis problem can be decreased when there is some information on the support of the sparse signal in the dictionary. They showed that the amount of this reduction is exactly the prior knowledge of frequencies in the support.

Vaswani et al. in [8], studied modified CS where a part of the signal support is known. They used weighted $\ell_{1}$ minimization to exploit this prior information and assigned zero weights to the known part. They obtained sufficient conditions for exact recovery and developed the so-called RegModCS algorithm that exploits prior knowledge of the signal. Jacques in [10], generalized the results of Vaswani et al. to the case of corrupted and noisy measurements.

Khajehnejad et al. in [5], investigated the non-uniform sparse model where the entries of the underlying sparse signal fall into a fixed number of partitions. Each partition has a different probability of being non-zero. Given these probabilities and using a Grassmann angle approach, they computed optimal weights in the case that the underlying signal divided into two subclasses. They used weighted $\ell_{1}$ minimization to recover the sparse signal. Their approach does not contain explicit strategies for more than two partitions.

Based on [6], and [11], Krishnaswamy et al. in [12], proposed a simpler and heuristic method to estimate the optimal weights. Unlike [5], their work can be extended to more than two sparsity partitions.

Oymak et al. in a different analysis in [13], considered the sparse signal with two partitions (non-uniform sparse model). To exploit prior knowledge, they used weighted $\ell_{1}$ minimization with each partition being assigned a weight. Using the ”escape through a mesh” lemma [14], they derived an upper bound for the minimum number of Gaussian measurements required for $\mathsf{P}_{1,\bm{w}}$ to succeed. In addition, they computed optimal weights by minimizing this upper bound with respect to the weights. Although they minimize the upper bound of the minimum number of measurements but they showed that this upper bound is tight asymptotically (as the ambient dimension goes to infinity) by relating the bound to the regularized normalized Minimum Mean Squared Error (MMSE) of a certain basis pursuit denoising problem (BPDN). Further, in [15], it has been shown that asymptotic normalized MMSE of BPDN equals to the asymptotic phase transition of Approximate Message Passing (AMP) algorithms.

Recently, Flinth in [16], generalized the results of Oymak et al. to the non-uniform setting with arbitrary number of partitions. He associated a weight with each partition and used the recent results of Amelunxen et al. in [17]. Amelunxen et al. had proved that the number of Gaussian measurements one needs to solve a convex program, is lower bounded by statistical dimension of a certain convex cone $\delta(C)$. They also calculated upper and lower bounds for $\delta(C)$. Flinth in [16] calculated the optimal weights by simultaneously minimizing the lower and upper bounds of $\delta(C)$ with respect to the weights.

I-B Applications

There are many important applications where in addition to inherent structures (entrywise sparsity in a dictionary, block sparsity and gradient sparsity), there exist some prior information about the signal of interest. In practice, the extraction of this prior information is realistic. As an example, one can investigate the statistic of used training data. Examples of applications with such information are listed below:

• •

Entrywise sparsity: In many digital signal processing applications, the desired discrete time signal is shown to be sparse in a specific dictionary. In [8], the recovery of a sparse signal is considered in which the support estimate in previous support (related to previous time instant) can improve the recovery performance by reducing the number of measurements. Natural images are often sparse in the discrete wavelet transform (DWT) as the dictionary. It is possible for wavelet image coders to incorporate some prior knowledge about the locations of large wavelet coefficients [18]. In fact, we have non-uniformity in the DWT and the largest coefficients are highly concentrated around zero. Also in many other applications, non-uniformity is present along sparsity in the frequency domain. One can exploit this feature and reduce number of measurements required for recovery using prior knowledge about the locations of the most dense parts in the frequency dictionary [9].

• •

Block sparsity: In block sparsity, the non-zero locations of the sparse signal appears in fixed blocks. There exist some applications with non-uniformity in the set of signal blocks. In DNA microarray [19], we have prior information that some blocks are most probable to include the non-zero elements. In computational neuroscience problems [20], the behavior of neurons exhibit non-uniform clustered responses. In [21], it is shown that non-uniform sampling problems over union of subspaces can be considered as block sparsity with non-uniform prior information on the blocks. In such cases, the general approach is to solve weighted $\ell_{1,2}$ optimization problem ( known as $\mathsf{P}_{1,2,\bm{w}}$) and assign larger weights to the blocks that are less likely to have non-zero entries.

• •

Gradient sparsity: Here, the signal of interest is sparse in the gradient dictionary. Conventional total variation (TV) semi norm assigns the same cost to smooth and non-smooth regions. However, natural images usually have non-uniform smoothness. Prior information in TV has long been investigated where we have some uncertainty about the smoothness of the 1- or 2-dimensional signals. In [22], non-uniform weight penalization is considered. In [23], weighted total variation optimization problem (known as $\mathsf{P}_{\mathrm{TV},\bm{w}}$) is used to consider non-uniform smoothness of the signal.

I-C Contributions

Unlike the prior works that only consider prior information on the signal entries in the identity dictionary, in Section II, we define a generalized framework that consider prior information on blocks, signal variations, or on the coefficients of the signal in a redundant dictionary, and extend the results of [16], and [13], to general signal models. Structures we cover in this paper, include sparse vectors in a specific possibly redundant dictionary, block sparse vectors, and piecewise constant signals. $\|\bm{\Omega}.\|_{0}$ (number of non-zero entries in the dictionary $\bm{\Omega}$), $\|.\|_{0,2}$ (number of blocks with non-zero $\ell_{2}$ norm), and $\|.\|_{\mathrm{NV}}$ (number of variations) promote the mentioned structures, respectively. The convex relaxed form of the above functions is $\|\bm{\Omega}.\|_{1}$, $\|.\|_{1,2}$ and $\|.\|_{\mathrm{TV}}$, respectively. Convex functions that exploit both the inherent structure and prior knowledge are $\|\bm{\Omega}.\|_{1,\bm{w}}$, $\|.\|_{1,2,\bm{w}}$ and $\|.\|_{\mathrm{TV},\bm{w}}$, respectively. Related convex optimization problems are called $\mathsf{P}_{1,\bm{\Omega},\bm{w}}$, $\mathsf{P}_{1,2,\bm{w}}$, and $\mathsf{P}_{\mathrm{TV},\bm{w}}$, respectively, which are introduced in Section II. The prior information in this paper, is the knowledge of partitions and probabilities. This setup exists in applications that multiple estimates of the support are available with different levels of confidence. In other words, these support estimates have different levels of accuracy. In this paper, we divide the structured signal555The structured signal may be block sparse, sparse in a dictionary, or smooth into some partitions and assign a weight to each partition. We compute optimal weights following the same strategy as in [16] and [13] regarding the given probability of partitions intersected by the support. However, our definition of support and partitions differ from those works due to their different structures. More precisely, with each inherent structure, we define partitions and support as follows:

Entrywise sparsity: The support is defined as the non-zero locations in the possibly redundant sparsity dictionary. In this case, we have the prior information that some fixed sets (partitions) in the dictionary, are intersected by the support with known probabilities. Throughout the paper, we refer to this kind of support as entrywise support denoted by $\mathcal{S}$. 2. 2.

Block sparsity: The support is defined as the blocks with non-zero $\ell_{2}$ norm. In this case, we have some sets of blocks with known probability that each set contributes to the support. Later on, we refer to this kind of support as block support $\mathcal{B}$. 3. 3.

Gradient sparsity: In this case, the support is defined as locations of signal variations. We refer to this kind of support as gradient support $\mathcal{S}_{g}$. The gradient support is divided into two sets: consecutive and individual support which refer to consecutive and individual variations, respectively. Some predefined sets in the gradient domain are intersected by the subsequent and individual supports with known probability.

Our main contributions are as follows:

Given the probability of partitions to be in the support, unique optimal weights will be computed in the three models mentioned above666Optimality of weights in block and gradient sparsity is in a different sense from entrywise sparsity in redundant dictionary as is made precise later.. 2. 2.

With optimal weights in an arbitrary redundant dictionary $\bm{\Omega}$ in $\mathsf{P}_{1,\bm{\Omega},\bm{w}}$, the required number of measurements to succeed, exactly equals the total number of measurements one needs to recover each partitioned sparse vector in the dictionary separately by solving $\mathsf{P}_{1}$. 3. 3.

With optimal weights on the blocks, the number of measurements required for successful recovery of a block sparse vector using $\mathsf{P}_{1,2,\bm{w}}$ with high probability, exactly equals the total number of measurements needed for recovery of each partition of blocks separately by solving $\mathsf{P}_{1,2}$. 4. 4.

With optimal weights on the signal variations, the required number of measurements for $\mathsf{P}_{\mathrm{TV},\bm{w}}$ to succeed with high probability equals the whole number of measurements one needs to recover each partitioned smooth vector separately by solving $\mathsf{P}_{\mathrm{TV}}$. 5. 5.

Explicit formulas are found for the number of measurements one needs to recover block sparse, smooth, and sparse vectors in a redundant dictionary.

This work includes the results of [16] and [13] as a special case of block sparsity with unit size blocks and entrywise sparsity with identity dictionary.

I-D Outline of the paper

The paper is organised as follows: Signal model and methodology are given in Section II. In Section III, basic concepts of conic integral geometry are reviewed. In Sections IV, V, and VI, entry-wise sparsity, block and gradient sparsity, are investigated, respectively. Numerical simulations that support the theory are presented in ̵ُSection VII. Finally, the paper is concluded in Section VIII.

I-E Notation

Throughout the paper, scalars are denoted by lowercase letters, vectors by lowercase boldface letters, and matrices by uppercase boldface letters. The $i$th element of the vector $\bm{x}$ is given either by ${x}(i)$ or $x_{i}$. $(\cdot)^{\dagger}$ denotes pseudo inverse. We reserve calligraphic uppercase letters for sets (e.g. $\mathcal{S}$). The cardinality of a set $\mathcal{S}$ is denoted by $|\mathcal{S}|$. $[n]$ refers to $\{1,...,n\}$. Furthermore, we write ${\mathcal{\bar{S}}}$ for the complement $[n]\setminus\mathcal{S}$ of a set $\mathcal{S}$ in $[n]$. For a matrix $\bm{X}\in\mathbb{R}^{m\times n}$ and a subset $\mathcal{S}\subseteq[n]$, the notation $\bm{X}_{\mathcal{S}}$ is used to indicate the column submatrix of $\bm{X}$ consisting of the columns indexed by $\mathcal{S}$. Similarly, for $\bm{x}\in\mathbb{R}^{n}$, $\bm{x}_{\mathcal{S}}$ is either the subvector in $\mathbb{R}^{|\mathcal{S}|}$ consisting of the entries indexed by $\mathcal{S}$, that is, $(\bm{x}_{S})_{i}=x_{j_{i}}~{}:~{}\mathcal{S}=\{j_{i}\}_{i=1}^{|\mathcal{S}|}$, or the vector in $\mathbb{R}^{n}$ which coincides with $\bm{x}$ on the entries in $\mathcal{S}$ and is zero on the entries outside $\mathcal{S}$. In this paper, $\bm{1}_{\mathcal{E}}$ denotes the indicator of the set $\mathcal{E}$. Nullspace and range of linear operators are denoted by $\mathrm{null}(\cdot)$, and $\mathrm{range}(\cdot)$, respectively. Hadamard product and Hadamard inverse on vectors are denoted by $\odot$ and $(\cdot)^{-1\odot}$ symbols, respectively. Given a vector $\bm{x}\in\mathbb{R}^{n}$ and a set $\mathcal{C}\subseteq\mathbb{R}^{n}$, denote the set obtained by scaling elements of $\mathcal{C}$ by elements of $\bm{x}$ by $\bm{x}\odot\mathcal{C}$. For a matrix $\bm{A}$, the operator norm is defined as $\|\bm{A}\|_{p\rightarrow q}=\underset{\|\bm{x}\|_{p}\leq 1}{\sup}\|\bm{Ax}\|_{q}$. For $\bm{x},\bm{y}\in\mathbb{R}^{n}$, $\bm{x}\leq\bm{y}$ denotes component-wise inequality while $\bm{x}<\bm{y}$ denotes component-wise inequality with strict inequality in at least one component. $\mathds{B}_{\epsilon}^{n}$ refers to the $\epsilon$-ball $\mathds{B}_{\epsilon}^{n}=\{\bm{x}\in\mathbb{R}^{n}:~{}\|\bm{x}\|_{2}\leq\epsilon\}$.

II Signal Model and Methodology

II-A Structured Signal Model

First, three models are defined as follow:

Definition II.1*.*

(Non-uniform Sparse Signal in Dictionary $\bm{\Omega}\in\mathbb{R}^{p\times n}$): Consider a vector $\bm{x}\in\mathbb{R}^{n}$ that is sparse in $\bm{\Omega}$ with support $\mathcal{S}$ and relative sparsity $\sigma=\frac{\|\bm{\Omega x}\|_{0}}{p}$. Let $\{\mathcal{P}_{i}\}_{i=1}^{L}$ be a partition of $[p]$. Each set $\mathcal{P}_{i}$ is associated with accuracy $\alpha_{i}=\frac{|\mathcal{P}_{i}\cap\mathcal{S}|}{|\mathcal{P}_{i}|}$ and relative size $\rho_{i}=\frac{|\mathcal{P}_{i}|}{p}$. A vector $\bm{x}\in\mathbb{R}^{n}$ is called a non-uniform sparse vector in the dictionary $\bm{\Omega}\in\mathbb{R}^{p\times n}$, if its sparsity pattern is defused along the partition $\{\mathcal{P}_{i}\}_{i=1}^{L}$. Number of non-zero entries of each set $\mathcal{P}_{i}$ is $\alpha_{i}\rho_{i}p$ such that the total sparsity in $\bm{\Omega}$ is $s=\sigma p=\sum_{i=1}^{L}\alpha_{i}\rho_{i}p$.

Definition II.2*.*

(Non-uniform Block Sparse Signal in $\mathbb{R}^{n}$): Consider a block sparse vector in $\bm{x}\in\mathbb{R}^{n}$ with the block support $\mathcal{B}$ that is divided into $q$ fixed blocks $\{\mathcal{V}_{b}\}_{b=1}^{q}\subset[n]$ of integer length $k=n/q$. Let $\{\mathcal{P}_{i}\}_{i=1}^{L}$ be a partition of the blocks $[q]$. To each set of blocks, accuracy $\alpha_{i}=\frac{|\mathcal{P}_{i}\cap\mathcal{B}|}{|\mathcal{P}_{i}|}$, and relative size $\rho_{i}=\frac{|\mathcal{P}_{i}|}{q}$ are assigned. A vector $\bm{x}\in\mathbb{R}^{n}$ is called a non-uniform block sparse vector with relative block sparsity $\sigma=\frac{\|\bm{x}\|_{0,2}}{q}$ if its sparsity pattern is defused along the blocks $[q]$. Number of non-zero blocks of each set $\mathcal{P}_{i}$ is $\alpha_{i}\rho_{i}q$ while the total block sparsity is $s=\sigma q=\sum_{i=1}^{L}\alpha_{i}\rho_{i}q$.

Definition II.3*.*

(Non-uniform Smooth Signal in $\mathbb{R}^{n}$): Consider a smooth vector $\bm{x}\in\mathbb{R}^{n}$ with gradient support $\mathcal{S}_{g}$ and relative smoothness $\sigma=\frac{\|\bm{x}\|_{NV}}{n-1}$. Let $\{\mathcal{P}_{i}\}_{i=1}^{L}$ be a partition of the signal variations $[n-1]$. Each set is associated with accuracy $\frac{|\mathcal{P}_{i}\cap\mathcal{S}_{g}|}{n-1}$ and relative size $\rho_{i}=\frac{|\mathcal{P}_{i}|}{n-1}$. A vector $\bm{x}\in\mathbb{R}^{n}$ is called a non-uniform smooth vector if its variation pattern is defused along $[n-1]$. Number of non-zero entries of each set $\mathcal{P}_{i}$ is $|\mathcal{P}_{i}\cap\mathcal{S}_{g}|$ such that the total gradient sparsity is $s=\sigma(n-1)=\sum_{i=1}^{L}|\mathcal{P}_{i}\cap\mathcal{S}_{g}|$.

In this paper, three inherent structures are considered:

Entrywise sparsity: Each of $L$ partitions is assigned a fixed weight. The related optimization problem becomes:

[TABLE]

in which, $\bm{w}=\sum_{i=1}^{L}\omega_{i}\bm{1}_{\mathcal{P}_{i}}\in\mathbb{R}_{+}^{p}$ and $(\bm{1}_{\mathcal{P}_{i}})_{j}=1:\forall j\in\mathcal{P}_{i}$ and [math] otherwise. 2. 2.

Block sparsity: Each of $L$ partitions is assigned a fixed weight. The related optimization problem will be:

[TABLE]

where, $\bm{w}=\sum_{i=1}^{L}\omega_{i}1_{\mathcal{P}_{i}}\in\mathbb{R}_{+}^{q}$. 3. 3.

Gradient sparsity: Each of $L$ partitions is assigned a fixed weight. The related optimization problem is:

[TABLE]

and, $\bm{w}=\sum_{i=1}^{L}\omega_{i}1_{\mathcal{P}_{i}}\in\mathbb{R}_{+}^{n-1}$

II-B Measurement Matrix

The common approach for sampling procedure is to draw a random measurement matrix $\bm{A}\in\mathbb{R}^{m\times n}$ consisting of i.i.d Gaussian entries. The authors in [17] show that the phase transition occurs when the measurement matrix $\bm{A}$ in $\mathsf{P}_{f}$ is standard normal with independent entries. However, the recent result of [24] shows that in the asymptotic case one can use a larger class of random measurements that have symmetric distribution and bounded moments. In this paper, we choose the entries of $\bm{A}$ from i.i.d standard Gaussian distribution.

II-C Methodology

For each structure defined in Section II-A , the number of measurements is calculated as a function of the weights ($w=\sum_{i=1}^{L}\omega_{i}1_{\mathcal{P}_{i}}$) and the optimal weights are found by minimizing the upper and lower bounds of statistical dimension. It’s shown that these bounds are asymptotically tight. The optimal weights obtained in the problems $\mathsf{P}_{1,\bm{\Omega},\bm{w}}$, $\mathsf{P}_{1,2,\bm{w}}$, and $\mathsf{P}_{\mathrm{TV},\bm{w}}$ are unique up to an irrelevant positive scaling.

III Conic Geometry

In this section, basic concepts of conic integral geometry to be used later are reviewed.

III-A Subdifferntial

Subdifferntial of a proper777An everywhere defined function taking values in $(-\infty,\infty]$ with at least one finite value in $(-\infty,\infty)$. convex function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}\cup\{\pm\infty\}$ at $\bm{x}\in\mathbb{R}^{n}$ is given by:

[TABLE]

Proposition III.1*.*

Let $f:\mathbb{R}^{n}\rightarrow\mathbb{R}\cup\{\pm\infty\}$ be a proper convex function that is $1$-homogenous i.e. $f(\alpha\bm{z})=|\alpha|f(\bm{z})~{}:\forall\alpha\in\mathbb{R}$ and sub-additive i.e. $f(\bm{x}+\bm{y})\leq f(\bm{x})+f(\bm{y})~{}~{}:\forall\bm{x},\bm{y}\in\mathbb{R}^{n}$, then we have a simpler form of subdifferntial given by:

[TABLE]

where, $f^{*}(\bm{z})=\underset{f(\bm{y})\leq 1}{\sup}\langle\bm{z},\bm{y}\rangle$ is the dual function of $f(\bm{z})$.

Proof.

By taking $\bm{y}=\bm{0}$ in (7), we have: $\langle\bm{z},\bm{x}\rangle\geq f(\bm{x})-f(\bm{0})$ which with $f(\bm{0})={0}$ in mind due to the homogeneity, $f^{*}(\bm{z})\geq 1$. Also, by taking $\bm{y}=\bm{v}+\bm{x}$ in (7) and taking superemum from both sides under the condition $f(\bm{v})=1$ we have:

[TABLE]

where we used sub-additivity of $f$. Hence, we have $f^{*}(\bm{z})=1$ and subsequently $\langle\bm{z},\bm{x}\rangle=f(\bm{x})$.

On the other hand, if we have $\bm{z}$ such that $f^{*}(\bm{z})=1$ and $\langle\bm{z},\bm{x}\rangle=f(\bm{x})$, then for each $\bm{y}\in\mathbb{R}^{n}$:

[TABLE]

∎

III-B Descent Cones and Normal Cones

The descent cone of a proper convex function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}\cup\{\pm\infty\}$ at point $\bm{x}\in\mathbb{R}^{n}$ is the set of directions from $\bm{x}$ that do not increase $f$ at least for one step size:

[TABLE]

The descent cone of a convex function is a convex set. The normal cone of a convex function at $\bm{x}\in\mathbb{R}^{n}$ is defined as the polar of the descent cone defined by:

[TABLE]

There is a famous duality [25] between decent cone and subdifferntial of a convex function given by:

[TABLE]

III-C Gaussian Width and Statistical Dimension

Definition III.1*.*

Gaussian Width[11]: The Gaussian width of a set $\mathcal{C}\subset\mathbb{R}^{n}$ is defined as:

[TABLE]

where, $\bm{g}\in\mathbb{R}^{n}$ is a vector with independent and identically distributed (i.i.d) standard normal entries.

Definition III.2*.*

Statistical Dimension[17]: Let $\mathcal{C}\subseteq\mathbb{R}^{n}$ be a convex closed cone. Statistical dimension of $\mathcal{C}$ is defined as:

[TABLE]

where, $\mathcal{P}_{\mathcal{C}}(\bm{x})$ is the projection of $\bm{x}\in\mathbb{R}^{n}$ onto the set $\mathcal{C}$ defined as: $\mathcal{P}_{\mathcal{C}}(\bm{x})=\underset{\bm{z}\in\mathcal{C}}{\arg\min}\|\bm{z}-\bm{x}\|_{2}$.

Statistical dimension generalizes the concept of dimension of subspaces to the class of convex cones.

III-D Linear Inverse Problems and Optimality Condition

In [17], it is proved that any random convex optimization problem undergoes a phase transition as number of measurements increased. The location of the transition is determined by the statistical dimension of descent cone of $f$ at $\bm{x}\in\mathbb{R}^{n}$ i.e. $\delta(\mathcal{D}(f,\bm{x}))$. Width of the transition from failure to success of $\mathsf{P}_{f}$ is $\mathcal{O}(\sqrt{n})$ measurements.

[TABLE]

First we express the optimality condition for $\mathsf{P}_{f}$ in the noise-free case.

Proposition III.2*.*

[11, Proposition 2.1] Optimality condition: Let $f$ be a proper convex function. The vector $\bm{x}\in\mathbb{R}^{n}$ is the unique optimal point of $\mathsf{P}_{f}$ if $\mathcal{D}(f,\bm{x})\cap\mathrm{null}(\bm{A})=\{\bm{0}\}$.

The next theorem determines number of measurements needed for successful recovery in $\mathsf{P}_{f}$ for any proper convex function $f$.

Theorem III.3*.*

[17, Theorem 2]: Let $f:\mathbb{R}^{n}\rightarrow\mathbb{R}\cup\{\pm\infty\}$ be a proper convex function and $\bm{x}\in\mathbb{R}^{n}$ a fixed sparse vector. Suppose that $m$ independent Gaussian linear measurements are taken from $\bm{x}$ collected in a vector $\bm{y}=\bm{Ax}\in\mathbb{R}^{m}$. Then for a given tolerance $\eta\in[0,1]$ if $m\geq\delta(\mathcal{D}(f,\bm{x}))+\sqrt{8\log(\frac{4}{\eta})n}$ we have: $\mathds{P}(\mathcal{D}(f,\bm{x})\cap\mathrm{null}(\bm{A})=\{\bm{0}\})\geq 1-\eta$. On the other hand, if $m\leq\delta(\mathcal{D}(f,\bm{x}))-\sqrt{8\log(\frac{4}{\eta})n}$ then, $\mathds{P}(\mathcal{D}(f,\bm{x})\cap\mathrm{null}(\bm{A})=\{\bm{0}\})\leq\eta$.

Also in [17], an error bound for the statistical dimension is given by the following theorem.

Theorem III.4*.*

[17, Theorem 4.3] For any $\bm{x}\in\mathbb{R}^{n}\setminus\{\bm{0}\}$:

[TABLE]

IV Entrywise sparsity in a dictionary

In this section, the following questions are investigated about a non-uniform sparse model in $\bm{\Omega}\in\mathbb{R}^{p\times n}$:

How many measurements one needs to recover an $s$-sparse vector in $\bm{\Omega}\in\mathbb{R}^{p\times n}$ by solving $\mathsf{P}_{1,\bm{\Omega}}$ and $\mathsf{P}_{1,\bm{\Omega},\bm{w}}$? 2. 2.

What is the optimal choice of weights in $\mathsf{P}_{1,\bm{\Omega},\bm{w}}$ given extra prior information?

IV-A Number of Measurements for successful Recovery

First we state the following theorem.

Theorem IV.1*.*

Let $\bm{x}\in\mathbb{R}^{n}$ be a fixed $s$-sparse vector in the dictionary $\bm{\Omega}$ (i.e. $\|\bm{\Omega x}\|_{0}\leq s$). Suppose that $m$ independent Gaussian linear measurements are taken from $\bm{x}$ collected in a vector $\bm{y}=\bm{Ax}\in\mathbb{R}^{m}$. Then for a given tolerance $\eta\in[0,1]$, if

[TABLE]

then the program $\mathsf{P}_{1,\bm{\Omega}}$ succeeds with probability at least $1-\eta\exp(-\frac{\mu^{2}}{8n})$. In other words, we have:

[TABLE]

where $\kappa(\bm{\Omega})=\frac{\sigma_{max}(\bm{\Omega})}{\sigma_{min}(\bm{\Omega})}$ is the condition number of the matrix $\bm{\Omega}$ and $\mu=\kappa^{2}(\bm{\Omega})\delta(\mathcal{D}(\|.\|_{1},\bm{\Omega x}))-\delta(\mathcal{D}(\|\bm{\Omega}.\|_{1},\bm{x}))+1$.

In what follows in this section, we denote the normalized number of measurements required to solve $\mathsf{P}_{1,\bm{\Omega}}$ and $\bm{P}_{1,\bm{\Omega},\bm{w}}$ with probability $1-\eta\exp(-\frac{\mu^{2}}{8n})$ by $m_{p,s}$ and $m_{p,s,\bm{w}}$, respectively which are defined as:

[TABLE]

[TABLE]

Theorem IV.1 is proved in Appendix B. Meanwhile in the proof, we derive an upper bound for $\delta(\mathcal{D}(\|\bm{\Omega}.\|_{1},\bm{x}))$ given in the following.

Proposition IV.2*.*

Statistical dimension of the descent cone of $\|\bm{\Omega}.\|_{1}$ at $\bm{x}\in\mathbb{R}^{n}$ is upper bounded by statistical dimension of the descent cone of $\|\cdot\|_{1}$ at $\bm{\Omega x}$ up to a positive scaling depending on $\bm{\Omega}$. Precisely, we have:

[TABLE]

Proof of Proposition IV.2 is given in Appendix A. In the following, we obtain upper bounds for number of measurements required for $\mathsf{P}_{1,\bm{\Omega}}$ and $\mathsf{P}_{1,\bm{\Omega},\bm{w}}$ to succeed.

Lemma IV.3*.*

Let $\bm{x}\in\mathbb{R}^{n}$ be an $s$-sparse vector in $\bm{\Omega}\in\mathbb{R}^{p\times n}$. Then an upper bound for normalized number of measurements required for $\mathsf{P}_{1,\bm{\Omega}}$ to succeed (i.e. $m_{p,s}$) is given by:

[TABLE]

with $\Psi_{t}(\sigma)$ defined as:

[TABLE]

where $\phi(z):=\sqrt{\frac{2}{\pi}}\int_{z}^{\infty}(u-z)^{2}\exp(-\frac{u^{2}}{2})du$.

Lemma IV.4*.*

Let $\bm{x}\in\mathbb{R}^{n}$ be a non-uniform $s$-sparse vector in $\bm{\Omega}\in\mathbb{R}^{p\times n}$ with parameters $\{\rho_{i}\}_{i=1}^{L}$ and $\{\alpha_{i}\}_{i=1}^{L}$. Then an upper bound for normalized number of measurements required for $\mathsf{P}_{1,\bm{\Omega},\bm{w}}$ with $\bm{w}=\sum_{i=1}^{L}\omega_{i}1_{\mathcal{P}_{i}}$ to succeed (i.e. $m_{p,s,\bm{w}}$) is given by:

[TABLE]

with $\Psi_{t,\bm{w}}(\sigma,\bm{\rho},\bm{\alpha})$ defined as:

[TABLE]

Proof.

By considering (19), we only need to find an upper bound for $\delta(\mathcal{D}(\|.\|_{1,\bm{w}},\bm{\Omega x}))/p$ defined as $\Psi_{t,\bm{w}}(\sigma,\bm{\rho},\bm{\alpha})$. By definition of the statistical dimension in (15), the fact that infimum of an affine function is concave and Jensen’s inequality, we have:

[TABLE]

By (25), we define our upper bound for $m_{p,s,\bm{w}}$ (i.e. $\hat{m}_{p,s,\bm{w}}$) which is given by:

[TABLE]

The next step is to calculate $\Psi_{t,\bm{w}}(\sigma,\bm{\rho},\bm{\alpha})$. For this purpose, $\partial\|.\|_{1,\bm{w}}(\bm{\Omega x})$ should be computed. From Proposition III.1, we have:

[TABLE]

[TABLE]

where in (IV-A), the inequality comes from Hölder’s inequality with equality when

[TABLE]

From the first part of (27), we find that $\bm{z}_{\mathcal{S}}=\bm{w}_{\mathcal{S}}\odot sgn(\bm{\Omega x})$ and from the second part together with (IV-A), we find out that $|w_{i}^{-1}z_{i}|\leq 1$. Hence,

[TABLE]

The distance between dilated subdifferntial of the descent cone of $\ell_{1,\bm{w}}$ norm at $\bm{\Omega x}$ and a standard Gaussian vector $\bm{g}$ is given by:

[TABLE]

where, $(a)_{+}:=\max\{a,0\}$ and triangle inequality is used in the second part. The expected value of (IV-A) is:

[TABLE]

where, $\zeta=|g_{i}|$ is distributed as a folded normal variable. Also:

[TABLE]

where in the third line, order of integration was changed. Thus, (IV-A) reads:

[TABLE]

By normalizing to the ambient dimension $p$ and incorporating prior information using $\bm{w}=\sum_{i=1}^{L}\omega_{i}1_{\mathcal{P}_{i}}$ we reach:

[TABLE]

where in the last line above, we benefited the fact that $\sigma=\sum_{i=1}^{L}\rho_{i}\alpha_{i}$. ∎

Proof of Lemma IV.3.

Regarding (18), we find an upper bound for $\delta(\mathcal{D}(\|.\|_{1},\bm{\Omega x}))/p$ which we denote by $\Psi_{t}(\sigma)$. The procedure is exactly the same as the proof of Lemma IV.4 with the weights set to one i.e. $\bm{w}=\bm{1}\in\mathbb{R}^{p}$. In fact, we have: $\Psi_{t}(\sigma)=\Psi_{t,\bm{w}}(\sigma,\bm{\rho},\bm{\alpha})$. By replacing $\bm{w}=\bm{1}\in\mathbb{R}^{p}$ in (35) and (26), we reach $\Psi_{t}(\sigma)$ and $\hat{m}_{p,s}$ in Lemma IV.3. ∎

In the following propositions, using the same technique as [17], we find out that the obtained upper bounds in Lemmas IV.3 and IV.4 are asymptotically tight.

Proposition IV.5*.*

. Normalized number of Gaussian linear measurements required for $\mathcal{P}_{1,\bm{\Omega}}$ to succeed (i.e. $m_{p,s}$) satisfies the following error bound.

[TABLE]

Proof.

By the error bound in (17) and (31) with $\bm{w}=\bm{1}\in\mathbb{R}^{p}$, we obtain the numerator of the error bound (17) as:

[TABLE]

Also, for the denominator we have:

[TABLE]

The error bound in (17) depends only on $\mathcal{D}(\|.\|_{1},\bm{\Omega x})$. Further, regarding (13), $\mathcal{D}(\|.\|_{1},\bm{\Omega x})$ depends only on $sgn(\bm{\Omega x})$ not the magnitudes of $\bm{\Omega x}$. So we can choose a vector

[TABLE]

with $sgn(\bm{z})=sgn(\bm{\Omega x})$ to have equality in (38). Therefore, the error in obtaining $\Psi_{t}(\sigma)$ is at most $\frac{1}{p}\frac{2\sqrt{p}}{\sqrt{s}}=\frac{2}{\sqrt{sp}}$. Further, the error of $\hat{m}_{p,s}$ in (21) is at most $\frac{2\kappa^{2}(\bm{\Omega})}{\sqrt{sp}}$. ∎

Proposition IV.6*.*

Normalized number of Gaussian linear measurements required for $P_{1,\bm{\Omega},\bm{w}}$ to successfully recover a non-uniform $s$-sparse vector in $\bm{\Omega}\in\mathbb{R}^{p\times n}$ with parameters $\{\rho_{i}\}_{i=1}^{L}$ and $\{\alpha_{i}\}_{i=1}^{L}$ (i.e. $m_{p,s,\bm{w}}$) satisfies the following error bound.

[TABLE]

It i interesting that the error bound in Proposition IV.2 is a special case of the error bound of Proposition IV.6 where one has $s$ sets with size $\frac{p}{L}$ and knows with probability $\frac{L}{p}$ that each set contributes to the entrywise support.

Proof.

By using (17) and (31), with $f(\bm{x}):=\|\bm{\Omega x}\|_{1,\bm{w}}$ and $\bm{w}=\sum_{i=1}^{L}\omega_{i}1_{\mathcal{P}_{i}}$, the numerator is given by:

[TABLE]

Also, for the denominator we have:

[TABLE]

where the first inequality comes from Cauchy Schwartz inequality. With the same justification as in the proof of Proposition IV.6 and the fact that $\mathcal{D}(\|.\|_{1,\bm{w}},\bm{\Omega x})$ depends only on $sgn(\bm{\Omega x})$, a vector

[TABLE]

with $sgn(\bm{z})=sgn(\bm{\Omega x})$ can be chosen to have equality in (43). Therefore, the error of obtaining the upper bound of $\delta(\mathcal{D}(\|.\|_{1,\bm{w}},\bm{\Omega x}))/p$ i.e. $\Psi_{t,\bm{w}}(\sigma,\bm{\rho},\bm{\alpha})$ is

[TABLE]

in which, the last inequality follows from the facts that $|\mathcal{P}_{i}\cap\mathcal{S}|\geq 1$, $|\mathcal{P}_{i}|\leq\frac{p}{L}$ for at least one $i\in[p]$ and thus $\underset{{i\in[p]}}{\min}\frac{|\mathcal{P}_{i}\cap\mathcal{S}|}{|\mathcal{P}_{i}|}\geq\frac{L}{p}$. Further the error in obtaining $\hat{m}_{p,s,\bm{w}}$ in (26) is at most $\frac{2\kappa^{2}(\bm{\Omega})}{\sqrt{pL}}$. ∎

IV-B Optimal Weights

Infimum of (42) gives:

[TABLE]

where $\bm{D}:=[\bm{1}_{\mathcal{P}_{1}},...,\bm{1}_{\mathcal{P}_{L}}]$. We call the weight $\bm{\omega}^{*}=\underset{\bm{\omega}\in\mathbb{R}_{+}^{L}}{\arg\min}~{}~{}\hat{m}_{p,s,\bm{D}\bm{\omega}}\in\mathbb{R}_{+}^{L}$ optimal since it asymptotically minimizes the number of measurements required for $\mathsf{P}_{1,\bm{\Omega},\bm{D\omega}}$ to succeed. Before showing that there exist unique optimal weights for $\mathsf{P}_{1,\bm{\Omega},\bm{D\omega}}$, we state the following lemma.

Lemma IV.7*.*

Let $\mathcal{C}:=\partial\|.\|_{1}(\bm{\Omega x})$. Suppose that $\mathcal{C}$ does not contain the origin. In particular, it is compact and there are upper and lower bounds that satisfy $1\leq\|\bm{z}\|_{2}\leq\sqrt{p}$ for all $\bm{z}\in\mathcal{C}$. Also denote the standard normal vector by $\bm{g}\in\mathbb{R}^{p}$. Consider the function

[TABLE]

The function $J$ is strictly convex, continuous at $\bm{\nu}\in\mathbb{R}_{+}^{L}$ and differentiable for $\bm{\nu}\in\mathbb{R}_{++}^{L}$. More over, there exists a unique point that minimize $J$.

Proof.

Continuity in bounded points. We must show that sufficiently small changes in $\bm{\nu}$ result in arbitrary small changes in $J(\bm{\nu})$. By definition of $J_{\bm{g}}(\bm{\nu})$, we have:

[TABLE]

where, we used the fact that,

[TABLE]

and,

[TABLE]

As a consequence, we obtain:

[TABLE]

Since $\|\bm{\nu}\|_{1}$ is bounded, continuity holds.

Convexity. Let $\bm{\nu}~{},\tilde{\bm{\nu}}\in\mathbb{R}_{+}^{L}$ and $\theta\in[0,1]$ with $\bm{\upsilon}=\bm{D\nu}$ and $\tilde{\bm{\upsilon}}=\bm{D}\tilde{\bm{\nu}}$. Then we have:

[TABLE]

Since otherwise we have:

[TABLE]

By taking the infimum over $\bm{z},\tilde{\bm{z}}\in\mathcal{C}$, we reach a contradiction. We proceed to prove convexity of $\mathrm{dist}(\bm{g},(\bm{D\nu})\odot\mathcal{C})$.

[TABLE]

Since this holds for any $\epsilon$ and $\tilde{\epsilon}$, $\mathrm{dist}(\bm{g},(\bm{D\nu})\odot\mathcal{C})$ is a convex function. As the square of a non-negative convex function is convex, $J_{\bm{g}}(\bm{\nu})$ is a convex function. At last, the function $J(\bm{\nu})$ is the average of convex functions, hence is convex. In (IV-B), the first inequality comes from the fact that $\forall\bm{z}_{1},\bm{z}_{2}\in\mathcal{C}~{}~{}\exists\bm{z}\in\mathcal{C}$:

[TABLE]

To verify (IV-B), note that:

[TABLE]

Then, setting $z(j)=1$ we reach a contradiction. In the second inequality in (IV-B), we used triangle inequality of norms. The third inequality uses (IV-B).

Strict convexity. We show strict convexity by contradiction. If $J(\bm{\nu})$ were not strictly convex, there would be vectors $\bm{\nu},\tilde{\bm{\nu}}\in\mathbb{R}_{+}^{L}$ with $\bm{\upsilon}=\bm{D}\bm{\nu},\tilde{\bm{\upsilon}}=\bm{D}\tilde{\bm{\nu}}$ and $\theta\in(0,1)$ such that,

[TABLE]

For each $\bm{g}$ in (67) the left-hand side is smaller than or equal to the right-hand side. Therefore, in (67), $J_{\bm{g}}(\theta\bm{\nu}+(1-\theta)\tilde{\bm{\nu}})$ and $\theta J_{\bm{g}}(\bm{\nu})+(1-\theta)J_{\bm{g}}(\tilde{\bm{\nu}})$ are almost surely equal (except at a measure zero set) with respect to Gaussian measure. Moreover, we have:

[TABLE]

where, the first inequality comes from (IV-B) and the second inequality stems from the strict convexity of $\|.\|_{2}^{2}$. From (IV-B), it is easy to verify that the set $\bm{\nu}\odot\mathcal{C}$ is a convex set. The distance to a convex set e.g. $\mathcal{E}$ i.e. $\mathrm{dist}(\bm{g},\mathcal{E})$ is a 1-lipschitz function (i.e. $|\mathrm{dist}(\bm{g},\mathcal{E})-\mathrm{dist}(\tilde{\bm{g}},\mathcal{E})|\leq\|\bm{g}-\tilde{\bm{g}}\|_{2}~{}:~{}\forall~{}\bm{g},\tilde{\bm{g}}\in\mathbb{R}^{p}$) and hence continuous with respect to $\bm{g}$. Therefore, $J_{\bm{g}}(\bm{\nu})$ is continuous with respect to $\bm{g}$. So there exist an open ball around $\bm{g}=\bm{0}\in\mathbb{R}^{p}$ that similar to (IV-B), we may write the following relation for some $\epsilon>0$

[TABLE]

$\mathds{B}_{\epsilon}^{p}$ is not a measure zero set. Thus, the above statement contradicts (67) and hence we have strict convexity. Continuity along with convexity of $J$ implies that $J$ is convex on the whole domain $\bm{\nu}\in\mathbb{R}_{+}^{L}$.

Differentiability. The function $J_{\bm{g}}(\bm{\nu})$ is continuously differentiable and the gradient for $\bm{\nu}\in\mathbb{R}_{++}^{L}$ is:

[TABLE]

Continuity of $\frac{\partial J_{\bm{g}}(\bm{\nu})}{\partial\bm{\nu}}$ at $\bm{\nu}\in\mathbb{R}_{+}^{L}$ stems from the fact that the projection onto a convex set is continuous. For each compact set $\mathcal{I}\subseteq\mathbb{R}_{+}^{L}$ we have:

[TABLE]

where $\nu_{\max}:=\underset{i\in[L]}{\max}~{}\nu(i)$. Therefore, we have:

[TABLE]

where in the last equality, we used the Lebesgue’s dominated convergence theorem.

Attainment of the minimum. Suppose that $\bm{\nu}>{\|\bm{g}\|_{2}}\bm{1}_{L\times 1}$. With this assumption we may write:

[TABLE]

where in (IV-B), $\nu_{\min}:=\underset{i\in[L]}{\min}~{}\nu(i)$. By squaring (IV-B), we reach:

[TABLE]

Using the relation $\mathds{E}\|\bm{g}\|_{2}\geq\frac{p}{\sqrt{p+1}}$ ([26, Proposition 8.1.]) and Marcov’s inequality we obtain:

[TABLE]

Then we reach:

[TABLE]

where in (IV-B), the first inequality stems from total probability theorem, the second inequality comes from (74). From (IV-B), we find out that $J(\bm{\nu})>J(\bm{0})$ when $\bm{\nu}>(2^{\frac{1}{4}}+1){\sqrt{p}}\bm{1}_{L\times 1}$. Therefore, the unique minimizer of the function $J$ must occur in the interval $[\bm{0},(2^{\frac{1}{4}}+1){\sqrt{p}}\bm{1}_{L\times 1}]$. ∎

Proposition IV.8*.*

Let $\bm{x}\in\mathbb{R}^{p}$ be a non-uniform $s$-sparse vector in $\bm{\Omega}\in\mathbb{R}^{p\times n}$ with parameters $\{\rho_{i}\}_{i=1}^{L}$ and $\{\alpha_{i}\}_{i=1}^{L}$. Then there exist unique optimal weights $\bm{\omega}^{*}\in R_{+}^{L}$ that minimize $\hat{m}_{p,s,\bm{w}}$ with $\bm{w}=\bm{D\omega}\in\mathbb{R}^{p}$. Moreover, the optimal weights are obtained via the following integral equations.

[TABLE]

Proof.

Define $\mathcal{C}:=\partial\|.\|_{1}(\bm{\Omega x})$ and use Lemma IV.4 and (31) to obtain:

[TABLE]

where, $\Psi_{t,\bm{D\omega}}(\sigma,\bm{\rho},\bm{\alpha})$ is defined in (24). Also, we used a change of variable $\bm{\upsilon}=t\bm{w}$ to convert the multivariate optimization problem to a single variable optimization problem. Thus, $J(\bm{\nu})$ is obtained via the following equation:

[TABLE]

As proved in Lemma IV.4, $J(\bm{\nu})$ is continuous and strictly convex. Thus, the unique minimizer can be obtained using $\nabla J(\bm{\nu})=\bm{0}$ which leads to:

[TABLE]

∎

Theorem IV.9*.*

Let $\bm{x}\in\mathbb{R}^{n}$ be a non-uniform $s$-sparse vector in $\bm{\Omega}\in\mathbb{R}^{p\times n}$ with parameters $\{\rho_{i}\}_{i=1}^{L}$ and $\{\alpha_{i}\}_{i=1}^{L}$. Then, number of measurements required for $\mathsf{P}_{1,\bm{\Omega},\bm{D}\bm{\omega}^{*}}$ is exactly equals the whole number of measurements required for $\mathsf{P}_{1}$ to recover each $\{(\bm{\Omega x})_{\mathcal{P}_{i}}\in\mathbb{R}^{p}\}_{i=1}^{L}$ separately up to an asymptotically negligible error term.

Proof.

As previously defined in (23), with optimal weights, the upper bound for normalized number of measurements required for $\mathsf{P}_{1,\bm{\Omega},\bm{D}\bm{\omega}^{*}}$ to succeed is given by:

[TABLE]

The expression in the bracket is exactly the upper bound for normalized number of measurements required for successful recovery of $(\bm{\Omega x})_{\mathcal{P}_{i}}\in\mathbb{R}^{p}$ using $\mathsf{P}_{1,\bm{\Omega}}$ i.e. $\hat{m}_{p,\|(\bm{\Omega x})_{\mathcal{P}_{i}}\|_{0}}$. Thus, regarding the error bounds obtained in Propositions IV.5 and IV.6, the relation between $m_{p,s,\bm{D}\bm{\omega}^{*}}$ and $m_{p,\|(\bm{\Omega x})_{\mathcal{P}_{i}}\|_{0}}$ is given by:

[TABLE]

∎

V Block Sparsity

In this section, the following questions are investigated about a non-uniform block sparse model:

How many measurements is required for $\mathsf{P}_{1,2}$ and $\mathsf{P}_{1,2,\bm{w}}$ to successfully recover an $s$-block sparse vector from independent Gaussian linear measurements? 2. 2.

Given extra prior information, what is the optimal choice of weights in $\mathsf{P}_{1,2,\bm{w}}$?

In what follows in this section, we precisely answer these questions in two subsections.

V-A Number of Measurement for successful Recovery

In this subsection, regarding Theorem (III.3) for $\|.\|_{1,2}$ and $\|.\|_{1,2,\bm{w}}$, we denote normalized number of measurements required for $\mathsf{P}_{1,2}$ and $\mathsf{P}_{1,2,\bm{w}}$ to recover an $s$-block sparse vector with probability $1-\eta$ by $m_{q,s}$ and $m_{q,s,\bm{w}}$, respectively which are defined as:

[TABLE]

[TABLE]

In the following, we obtain upper bounds for number of measurements required for $\mathsf{P}_{1,2}$ and $\mathsf{P}_{1,2,\bm{w}}$ to succeed with probability $1-\eta$.

Lemma V.1*.*

Let $\bm{x}\in\mathbb{R}^{n}$ be an $s$-block sparse vector. Then an upper bound for normalized number of measurements required for $\mathsf{P}_{1,2}$ to succeed (i.e. $m_{q,s}$) is given by:

[TABLE]

with $\Psi_{t}(\sigma)$ defined as:

[TABLE]

where $k=\frac{n}{q}$, $\phi_{B}(z):=\int_{z}^{\infty}(u-z)^{2}u^{k-1}\exp(-\frac{u^{2}}{2})du$ and $\sigma:=\frac{\|\bm{x}\|_{0,2}}{q}$

Lemma V.2*.*

Let $\bm{x}\in\mathbb{R}^{n}$ be a non-uniform $s$-block sparse vector in $\mathbb{R}^{n}$ with parameters $\{\rho_{i}\}_{i=1}^{L}$ and $\{\alpha_{i}\}_{i=1}^{L}$. Then an upper bound for normalized number of measurements required for $\mathsf{P}_{1,2,\bm{w}}$ with $\bm{w}=\bm{D}\bm{\omega}~{}\in\mathbb{R}^{q}$ to succeed (i.e. $m_{q,s,\bm{w}}$) is given by:

[TABLE]

with $\Psi_{t,\bm{w}}(\sigma,\bm{\rho},\bm{\alpha})$ defined as:

[TABLE]

Proof.

With the same reasoning as in (25) and the definition of statistical dimension for $\mathcal{D}(\|.\|_{1,2,\bm{w}},\bm{x})$ we have:

[TABLE]

The next step is to calculate $\partial\|.\|_{1,2,\bm{w}}(\bm{x})$. From Proposition III.1, we have:

[TABLE]

From the first part of (89), we obtain:

[TABLE]

which then we reach $\bm{z}_{\mathcal{V}_{b}}=\frac{w_{b}\bm{x}_{\mathcal{V}_{b}}}{\|\bm{x}_{\mathcal{V}_{b}}\|_{2}}~{}:~{}\forall b\in\mathcal{B}$. To compute the dual of $\|.\|_{1,2,\bm{w}}$ i.e. $\|.\|_{1,2,\bm{w}}^{*}$, we have:

[TABLE]

where in, the first and second inequality comes from Hölder’s inequality with equality when

[TABLE]

From (V-A) and (89), we find out that $w_{b}^{-1}\|\bm{z}_{\mathcal{V}_{b}}\|_{2}\leq 1~{}:~{}\forall b\in[q]$. Hence,

[TABLE]

Now to calculate $\Psi_{t,\bm{w}}(\sigma,\bm{\rho},\bm{\alpha})$, regarding (94), we compute the distance of the dilated subdifferntial of descent cone of $\ell_{1,2,\bm{w}}$ norm at $\bm{x}\in\mathbb{R}^{n}$ from a standard Gaussian vector $\bm{g}\in\mathbb{R}^{n}$ which is given by:

[TABLE]

where we used triangle inequality in the second part. By taking expectation from both sides, we reach:

[TABLE]

where, $k=\frac{n}{q}$. and $\zeta^{2}:=\|\bm{g}_{\mathcal{V}_{b}}\|_{2}^{2}$ has chi-square distribution with $k$ degrees of freedom. Moreover,

[TABLE]

where in the third line, the order of integration is changed and in the forth line, a change of variable is used. As a consequence, (V-A) becomes:

[TABLE]

By normalizing to the number of blocks $q$ and incorporating block prior information using $\bm{w}=\bm{D\omega}\in\mathbb{R}^{n}$ we reach:

[TABLE]

where in the last line above, we benefited the fact that $\sigma=\sum_{i=1}^{L}\rho_{i}\alpha_{i}$. ∎

Proof of Lemma V.1.

We find an upper bound for $m_{q,s}$. The procedure is exactly the same as the proof of Lemma V.2 with the assigned weight to each block set to one i.e. $w=1\in\mathbb{R}^{q}$. In fact, we have: $\Psi_{t}(\sigma)=\Psi_{t,\bm{w}}(\sigma,\bm{\rho},\bm{\alpha})$. By replacing $\bm{w}=\bm{1}\in\mathbb{R}^{q}$ in (V-A) and (88), we reach $\hat{m}_{q,s}$ in Lemma V.1. ∎

In the following Propositions, we prove that the obtained upper bounds in Lemmas V.1 and V.2 for normalized number of measurements are asymptotically tight.

Proposition V.3*.*

. Normalized number of Gaussian linear measurements required for $\mathsf{P}_{1,2}$ to succeed (i.e. $m_{q,s}$) satisfies the following error bound.

[TABLE]

Proof.

By the error bound in (17) and also (94) with $\bm{w}=\bm{1}\in\mathbb{R}^{q}$, we obtain the numerator of the error bound (17) as:

[TABLE]

Also, for the denominator we have:

[TABLE]

The error bound in (17) for $\|.\|_{1,2}$ depends only on $\mathcal{D}(\|.\|_{1,2},\bm{x})$. Moreover, $\mathcal{D}(\|.\|_{1,2},\bm{x})$ only requires that $\|\bm{x}_{\mathcal{V}_{b}}\|_{2}=1~{}:~{}\forall b\in\mathcal{B}$. So a vector

[TABLE]

can be chosen to have equality in (102). Therefore, the error of obtaining $m_{q,s}$ is at most $\frac{1}{q}\frac{2\sqrt{q}}{\sqrt{s}}=\frac{2}{\sqrt{sq}}$. ∎

Proposition V.4*.*

Normalized number of Gaussian linear measurements required for $\mathsf{P}_{1,2,\bm{w}}$ to successfully recover a non-uniform $s$-block sparse vector in $\mathbb{R}^{n}$ with parameters $\{\rho_{i}\}_{i=1}^{L}$ and $\{\alpha_{i}\}_{i=1}^{L}$ (i.e. $m_{q,s,\bm{w}}$) satisfies the following error bound.

[TABLE]

It is interesting that the error bound in Proposition V.3 is a special case of the error bound of Proposition V.4 where one has $s$ sets of blocks with size $\frac{q}{L}$ and knows with probability $\frac{L}{q}$ that each set of blocks contributes to the block support.

Proof.

By the error bound (17) and (94), with $f(\bm{x})=\|\bm{x}\|_{1,2,\bm{w}}$ and $\bm{w}=\sum_{i=1}^{L}\omega_{i}\bm{1}_{\mathcal{P}_{i}}\in\mathbb{R}^{q}$, the numerator of (17) is given by:

[TABLE]

Also, for the denominator we have:

[TABLE]

where the first inequality in (107) comes from Cauchy Schwartz inequality. With the same justification as in the proof of Proposition V.3 and the fact that $\mathcal{D}(\|.\|_{1,2,\bm{w}},\bm{x})$ only requires that $\|\bm{x}_{\mathcal{V}_{b}}\|_{2}=w_{b}~{}:~{}\forall b\in\mathcal{B}$ a vector

[TABLE]

can be chosen to have equality in (107). Therefore, the error of obtaining the upper bound of $m_{q,s,\bm{w}}$ i.e. $\hat{m}_{q,s,\bm{w}}$ is:

[TABLE]

in which, the last inequality follows from the facts that $|\mathcal{P}_{i}\cap\mathcal{B}|\geq 1$, $|\mathcal{P}_{i}|\leq\frac{q}{L}$ for at least one $i\in[q]$ and thus $\underset{{i\in[q]}}{\min}\frac{|\mathcal{P}_{i}\cap\mathcal{B}|}{|\mathcal{P}_{i}|}\geq\frac{L}{q}$. Further, the error of $\hat{m}_{q,s,\bm{w}}$ in (88) is at most $\frac{2}{\sqrt{qL}}$ ∎

V-B Optimal Weights

Infimum of (106) gives:

[TABLE]

where $\bm{D}:=[\bm{1}_{\mathcal{V}_{1}},...,\bm{1}_{\mathcal{V}_{q}}]_{n\times q}[\bm{1}_{\mathcal{P}_{1}},...,\bm{1}_{\mathcal{P}_{L}}]_{q\times L}$. In weighted block sparsity, we call the weight $\bm{\omega}^{*}=\underset{\bm{\omega}\in\mathbb{R}_{+}^{L}}{\arg\min}~{}~{}\hat{m}_{q,s,\bm{D\omega}}\in\mathbb{R}_{+}^{L}$ optimal since it asymptotically minimizes number of measurements required for $\mathsf{P}_{1,2,\bm{w}}$ to succeed. Similar to previous section, before proving the uniqueness of optimal weights for $\mathsf{P}_{1,2,\bm{D\omega}}$, we state the following lemma.

Lemma V.5*.*

Let $\mathcal{C}:=\partial\|.\|_{1,2}(\bm{x})$. Suppose that $\mathcal{C}$ does not contain the origin. In particular, it is compact and there are upper and lower bounds that satisfy $1\leq\|\bm{z}\|_{2}\leq\sqrt{q}$ for all $\bm{z}\in\mathcal{C}$. Also denote the standard normal vector by $\bm{g}\in\mathbb{R}^{n}$. Consider the function

[TABLE]

The function $J$ is strictly convex, continuous at $\bm{\nu}\in\mathbb{R}_{+}^{L}$ and differentiable for $\bm{\nu}\in\mathbb{R}_{++}^{L}$. More over, there exists a unique point that minimize $J$.

Proof.

Continuity in bounded points. We must show that sufficiently small changes in $\bm{\nu}$ result in arbitrary small changes in $J(\bm{\nu})$. By definition of $J_{\bm{g}}(\bm{\nu})$, we have:

[TABLE]

where, we used the fact that,

[TABLE]

and,

[TABLE]

As a consequence, we obtain:

[TABLE]

Since $\|\bm{\nu}\|_{1}$ is bounded, continuity holds.

Convexity. Let $\bm{\nu}~{},\tilde{\bm{\nu}}\in\mathbb{R}_{+}^{L}$ and $\theta\in[0,1]$ with $\bm{\upsilon}=\bm{D\nu}$ and $\tilde{\bm{\upsilon}}=\bm{D}\tilde{\bm{\nu}}$. Then we have:

[TABLE]

Since otherwise we have:

[TABLE]

By taking the infimum over $\bm{z},\tilde{\bm{z}}\in\mathcal{C}$, we reach a contradiction. We proceed to prove convexity of $\mathrm{dist}(\bm{g},(\bm{D\nu})\odot\mathcal{C})$.

[TABLE]

Since this holds for any $\epsilon$ and $\tilde{\epsilon}$, $\mathrm{dist}(\bm{g},(\bm{D\nu})\odot\mathcal{C})$ is a convex function. As the square of a non-negative convex function is convex, $J_{\bm{g}}(\bm{\nu})$ is a convex function. At last, the function $J(\bm{\nu})$ is the average of convex functions, hence is convex. In (V-B), the first inequality comes from the fact that $\forall\bm{z}_{1},\bm{z}_{2}\in\mathcal{C}~{}~{}\exists\bm{z}\in\mathcal{C}$:

[TABLE]

To verify (V-B), we argue by contradiction.

[TABLE]

Then, by taking $\bm{z}_{\mathcal{V}_{d}}=\bm{e}_{i}\in\mathbb{R}^{k}$ for some $i\in[k]$, we reach a contradiction. In the second inequality in (V-B), we used triangle inequality of norms. The third inequality uses the relation (V-B).

Strict convexity. We show strict convexity by contradiction. If $J(\bm{\nu})$ were not strictly convex, there would be vectors $\bm{\nu},\tilde{\bm{\nu}}\in\mathbb{R}_{+}^{L}$ with $\bm{\upsilon}=\bm{D\nu},\tilde{\bm{\upsilon}}=\bm{D}\tilde{\bm{\nu}}$ and $\theta\in(0,1)$ such that,

[TABLE]

For each $\bm{g}$ in (129) the left-hand side is smaller than or equal to the right-hand side. Therefore, in (129), $J_{\bm{g}}(\theta\bm{\nu}+(1-\theta)\tilde{\bm{\nu}})$ and $\theta J_{\bm{g}}(\bm{\nu})+(1-\theta)J_{\bm{g}}(\tilde{\bm{\nu}})$ are almost surely equal (except at a measure zero set) with respect to Gaussian measure. Moreover, we have:

[TABLE]

where, the first inequality comes from (V-B) and the second inequality stems from the strict convexity of $\|.\|_{2}^{2}$. From (V-B), it is easy to verify that the set $\bm{\nu}\odot\mathcal{C}$ is a convex set. The distance to a convex set e.g. $\mathcal{E}$ i.e. $\mathrm{dist}(\bm{g},\mathcal{E})$ is a $1$-lipschitz function (i.e. $|\mathrm{dist}(\bm{g},\mathcal{E})-\mathrm{dist}(\tilde{\bm{g}},\mathcal{E})|\leq\|\bm{g}-\tilde{\bm{g}}\|_{2}~{}:~{}\forall~{}\bm{g},\tilde{\bm{g}}\in\mathbb{R}^{n}$) and hence continuous with respect to $\bm{g}$. Therefore, $J_{\bm{g}}(\bm{\nu})$ is continuous with respect to $\bm{g}$. So there exist an open ball around $\bm{g}=\bm{0}\in\mathbb{R}^{n}$ that similar to (V-B), we may write the following relation for some $\epsilon>0$

[TABLE]

The above statement contradicts with (129) and hence we have strict convexity. Continuity along with convexity of $J$ implies that $J$ is convex on the whole domain $\bm{\nu}\in\mathbb{R}_{+}^{L}$.

Differentiability. The function $J_{\bm{g}}(\bm{\nu})$ is continuously differentiable and the gradient for $\bm{\nu}\in\mathbb{R}_{++}^{L}$ is:

[TABLE]

Continuity of $\frac{\partial J_{\bm{g}}(\bm{\nu})}{\partial\bm{\nu}}$ at $\bm{\nu}\in\mathbb{R}_{+}^{L}$ stems from the fact that the projection onto a convex set is continuous. For each compact set $\mathcal{I}\subseteq\mathbb{R}_{+}^{L}$ we have:

[TABLE]

where $\nu_{\max}:=\underset{i\in[L]}{\max}~{}\nu(i)$. Therefore, we have:

[TABLE]

where in the last equality, we used the Lebesgue’s dominated convergence theorem.

Attainment of the minimum. Suppose that $\bm{\nu}\geq{\|\bm{g}\|_{2}}\bm{1}_{L\times 1}$. With this assumption we may write:

[TABLE]

where in (V-B), $\nu_{\min}:=\underset{i\in[L]}{\min}~{}\nu(i)$. By squaring (V-B), we reach:

[TABLE]

Using the relation $\mathds{E}\|\bm{g}\|_{2}\geq\frac{n}{\sqrt{n+1}}$ and Marcov’s inequality we obtain:

[TABLE]

Then we reach:

[TABLE]

where in (V-B), the first inequality stems from total probability theorem, the second inequality comes from (136). From (V-B), we find out that $J(\bm{\nu})>J(\bm{0})$ when $\nu>(2^{\frac{1}{4}}+1){\sqrt{n}}\bm{1}_{L\times 1}$. Therefore, the unique minimizer of the function $J$ must occur in the interval $[\bm{0},(2^{\frac{1}{4}}+1){\sqrt{n}}\bm{1}_{L\times 1}]$ ∎

Proposition V.6*.*

Let $\bm{x}$ be a non-uniform $s$-block-sparse vector in $\mathbb{R}^{n}$ with parameters $\{\rho_{i}\}_{i=1}^{L}$ and $\{\alpha_{i}\}_{i=1}^{L}$. Then there exist unique optimal weights $\bm{\omega}^{*}\in\mathbb{R}_{+}^{L}$ (up to a positive scaling) that minimize $\hat{m}_{q,s,\bm{D\omega}}$. Moreover, the optimal weights $\bm{\omega}^{*}\in\mathbb{R}_{+}^{L}$ are obtained via the following integral equations.

[TABLE]

Proof.

Define $\mathcal{C}:=\partial\|.\|_{1,2}(\bm{x})$ and use Lemma V.2 and IV.7 to obtain:

[TABLE]

where, $\Psi_{t,\bm{D\omega}}(\sigma,\bm{\rho},\bm{\alpha})$ is defined in (86). Also, we used a change of variable $\bm{\nu}=t\bm{\omega}$ to convert multivariate optimization problem to a single variable optimization problem. Thus, the function $J_{b}(\bm{\nu})$ is obtained via the following equation:

[TABLE]

By considering Lemma IV.7 and $\bm{D}:=[\bm{1}_{\mathcal{V}_{1}},...,\bm{1}_{\mathcal{V}_{q}}]_{n\times q}[\bm{1}_{\mathcal{P}_{1}},...,\bm{1}_{\mathcal{P}_{L}}]_{q\times L}$, the function $J_{b}(\bm{\nu})$ is continuous and strictly convex and thus the unique minimizer can be obtained using $\nabla J_{b}(\bm{\nu})=\bm{0}\in\mathbb{R}^{L}$ which leads to:

[TABLE]

∎

Theorem V.7*.*

Let $\bm{x}$ be a non-uniform $s$-block-sparse vector in $\mathbb{R}^{n}$ with parameters $\{\rho_{i}\}_{i=1}^{L}$ and $\{\alpha_{i}\}_{i=1}^{L}$. Then, number of measurements required for $\mathsf{P}_{1,2,\bm{D\omega}^{*}}$ is exactly equals the whole number of measurements required for $\mathsf{P}_{1,2}$ to recover each $\{\bm{x}_{\mathcal{P}_{i}}\in\mathbb{R}^{n}\}_{i=1}^{L}$ separately up to an asymptotically negligible error term.

Proof.

As previously defined in (86), with optimal weights, the upper bound for normalized number of measurements required for $\mathsf{P}_{1,2,\bm{D\omega}^{*}}$ to succeed is:

[TABLE]

The expression in the bracket is exactly the upper bound for normalized number of measurements required for successful recovery of $\bm{x}_{\mathcal{P}_{i}}\in\mathbb{R}^{n}$ using $\mathsf{P}_{1,2}$ i.e. $\hat{m}_{q,\|\bm{x}_{\mathcal{P}_{i}}\|_{0,2}}$. Thus, regarding the error bounds obtained in Propositions V.3 and V.4, the relation between $m_{q,s,\bm{D\omega}^{*}}$ and $m_{q,\|\bm{x}_{\mathcal{P}_{i}}\|_{0,2}}$ is given by:

[TABLE]

∎

VI Gradient Sparsity

In this section, the following questions are investigated about a non-uniform smooth model.

How many measurements one needs to recover an $s$-gradient sparse vector in $\mathbb{R}^{n}$ by solving $\mathsf{P}_{\mathrm{TV}}$ and $\mathsf{P}_{\mathrm{TV},\bm{w}}$? 2. 2.

What is the optimal choice of weights in $\mathsf{P}_{\mathrm{TV},\bm{w}}$ given extra prior information?

In what follows in this section, we precisely answer these questions in two subsections.

VI-A Number of Measurements for successful Recovery

In this subsection, regarding Theorem (III.3) for the functions $\|.\|_{\mathrm{TV}}$ and $\|.\|_{\mathrm{TV},\bm{w}}$, we denote normalized number of measurements required for $\mathsf{P}_{\mathrm{TV}}$ and $\mathsf{P}_{\mathrm{TV},\bm{w}}$ to recover an $s$-gradient sparse vector with probability $1-\eta$ by $m_{\mathrm{TV},s}$ and $m_{\mathrm{TV},\bm{w}}$, respectively which are defined as:

[TABLE]

[TABLE]

In the following, we obtain upper bounds for number of measurements required for $\mathsf{P}_{\mathrm{TV}}$ and $\mathsf{P}_{\mathrm{TV},\bm{w}}$ to succeed with probability $1-\eta$ where $\eta$ is the tolerance. In what follows in this section, $\bm{\Omega}_{d}\in\mathbb{R}^{{n-1}\times n}$ given below denotes the difference operator and $\bm{d}:=\bm{\Omega}_{d}\bm{x}$ is the difference $s$-sparse vector in $\mathbb{R}^{n-1}$.

[TABLE]

Lemma VI.1*.*

Let $\bm{x}\in\mathbb{R}^{n}$ be an $s$-gradient sparse vector. Then an upper bound for normalized number of measurements required for $\mathsf{P}_{\mathrm{TV}}$ to succeed (i.e. $m_{\mathrm{TV},s}$) is given by:

[TABLE]

with $\Psi_{t}(\sigma_{1},\sigma_{2},\sigma_{3},\sigma_{4})$ defined as:

[TABLE]

where,

[TABLE]

In (VI.1), $\mathcal{S}_{1}\cup\mathcal{S}_{2}$ and $\mathcal{S}_{3}\cup\mathcal{S}_{4}$ refer to consecutive and individual support, respectively.

Lemma VI.2*.*

Let $\bm{x}\in\mathbb{R}^{n}$ be a non-uniform gradient sparse vector. Then an upper bound for normalized number of measurements required for $\mathsf{P}_{\mathrm{TV},\bm{w}}$ to succeed with high probability (i.e. $m_{\mathrm{TV},\bm{w}}$) is given by:

[TABLE]

with $\Psi_{t,\bm{w}}(\bm{\alpha},\bm{\alpha}^{\prime},\bm{\beta},\bm{\beta}^{\prime},\bm{\gamma},\bm{\gamma}^{\prime},\bm{\varsigma},\bm{\varsigma}^{\prime},\xi,\breve{\xi})$ defined as:

[TABLE]

where,

[TABLE]

In (VI.2), $\mathcal{P}_{i}\cap\mathcal{S}_{i}^{\prime}$ denotes consecutive indices with each index located in a different partition.

Before proceeding, we state a chain-rule lemma about subdifferential of a convex function which is later required.

Lemma VI.3* ([27]).*

let $f:\mathbb{R}^{m}\rightarrow(-\infty,\infty]$ be a proper convex function and $\bm{\Omega}$ be a matrix. Consider $F(\bm{x})=f(\bm{\Omega x})$. If $\mathrm{range}(\bm{\Omega})\cap\mathrm{relint}(dom(f))\neq\varnothing$, then:

[TABLE]

Proof of Lemma VI.2.

By definition of the statistical dimension in (15) for the semi-norm function $\|.\|_{\mathrm{TV},\bm{w}}$ and Jensen’s inequality, we have:

[TABLE]

The next step is to calculate $\partial\|.\|_{\mathrm{TV},\bm{w}}(\bm{x})$. From Lemma VI.3, we have:

[TABLE]

where we used (31) with $\bm{\Omega}=\bm{I}_{d}\in\mathbb{R}^{n\times n}$. Now to calculate $\Psi_{t,\bm{w}}(\bm{\alpha},\bm{\alpha}^{\prime},\bm{\beta},\bm{\beta}^{\prime},\bm{\gamma},\bm{\gamma}^{\prime},\bm{\varsigma},\bm{\varsigma}^{\prime},\bm{\xi},\tilde{\bm{\xi}})$, regarding (VI-A), we compute the distance of the dilated subdifferntial of descent cone of weighted TV norm at $\bm{x}\in\mathbb{R}^{n}$ from a standard Gaussian vector $\bm{g}\in\mathbb{R}^{n}$ which is given by:

[TABLE]

where $\zeta_{1}=|g_{i}-tw_{i}sgn(d_{i})|$, $\zeta_{2}=|g_{i}+tw_{i-1}sgn(d_{i-1})|$ and we used triangle inequality in the last part. By taking expectation from both sides, we reach:

[TABLE]

where,

[TABLE]

and in (VI-A), the order of integration is changed together with a change of variable. Similarly, we have: $\mathds{E}(\zeta_{2}-tw_{i})_{+}^{2}=\phi_{1}(tw_{i-1},tw_{i})$. With the same reasoning as in (IV-A), we have: $\mathds{E}(|g_{i}|-tw_{i}-tw_{i-1})_{+}^{2}=\phi_{2}(t(w_{i}+w_{i-1}))$. As a consequence, (VI-A) becomes:

[TABLE]

By incorporating prior information and assigning weights to the sorted partitions $\{\mathcal{P}_{i}\}_{i=1}^{L}$ using $\bm{w}=\bm{D\omega}\in\mathbb{R}^{n-1}$ with $\bm{D}:=[\bm{1}_{\mathcal{P}_{1}},...,\bm{1}_{\mathcal{P}_{L}}]\in\mathbb{R}^{n-1\times L}$ we reach:

[TABLE]

By normalizing to $n-1$, we reach (149). ∎

Proof of Lemma VI.1.

By considering (144), we find an upper bound for $\delta(\mathcal{D}(\|.\|_{\mathrm{TV}},\bm{x}))/n$ which we denote by $\Psi_{t}(\sigma_{1},\sigma_{2},\sigma_{3},\sigma_{4})$. The procedure is exactly the same as the proof of Lemma VI.2 with the weights set to one i.e. $\bm{w}=\bm{1}\in\mathbb{R}^{n-1}$. In fact, we have: $\Psi_{t}(\sigma_{1},\sigma_{2},\sigma_{3},\sigma_{4})=\Psi_{t,\bm{1}}(\bm{\alpha},\bm{\alpha}^{\prime},\bm{\beta},\bm{\beta}^{\prime},\bm{\gamma},\bm{\gamma}^{\prime},\bm{\varsigma},\bm{\varsigma}^{\prime},\xi,\breve{\xi})$. By replacing $\bm{w}=\bm{1}\in\mathbb{R}^{n-1}$ in (VI-A) and (VI-A), we reach $\Psi_{t}(\sigma_{1},\sigma_{2},\sigma_{3},\sigma_{4})$ and $\hat{m}_{\mathrm{TV},s}$ in Lemma VI.1. ∎

In the following propositions, we prove that the obtained upper bounds in Lemma’s VI.1 and VI.2, are asymptotically tight.

Proposition VI.4*.*

. Normalized number of Gaussian linear measurements required for $\mathsf{P}_{\mathrm{TV}}$ to succeed (i.e. $m_{\mathrm{TV},s}$) satisfies the following error bound.

[TABLE]

Proof.

By using the error bound (17) and (VI-A) with $\bm{w}=\bm{1}\in\mathbb{R}^{n-1}$, we obtain the numerator of the error bound (17) as given by:

[TABLE]

Also, the denominator can be lower bounded as below:

[TABLE]

The error bound in (17) depends only on $\mathcal{D}(\|.\|_{\mathrm{TV}},\bm{x})$ and $\partial\|.\|_{\mathrm{TV}}(\bm{x})$. Further, $\partial\|.\|_{\mathrm{TV}}(\bm{x})$ depends only on $sgn(\bm{d})$ not the magnitudes of $\bm{d}$. So a vector

[TABLE]

with $sgn(\bm{z})=sgn(\bm{d})$ can be chosen to have equality in the last inequality in (164). Therefore, the error of obtaining $\hat{m}_{\mathrm{TV},s}$ in (146) is at most $\frac{2\kappa(\bm{\Omega}_{d})}{\sqrt{(n-1)s}}$. ∎

Proposition VI.5*.*

Normalized number of Gaussian linear measurements required for $\mathsf{P}_{\mathrm{TV},\bm{w}}$ to successfully recover a non-uniform gradient sparse vector in $\mathbb{R}^{n}$ with parameters $\{\rho_{i}\}_{i=1}^{L}$ and $\{\alpha_{i},\alpha_{i}^{\prime},\beta_{i},\beta_{i}^{\prime},\gamma_{i},\gamma_{i}^{\prime},\varsigma_{i},\varsigma_{i}^{\prime},\xi_{i},\breve{\xi}_{i}\}_{i=1}^{L}$ (i.e. $m_{\mathrm{TV},\bm{w}}$) satisfies the following error bound.

[TABLE]

Proof.

By using the error bound (17) with $f(\bm{x}):=\|\bm{x}\|_{TV,\bm{w}}$, the subdifferntial (VI-A), and $\bm{w}=\sum_{i=1}^{L}\omega_{i}\bm{1}_{\mathcal{P}_{i}}$, the numerator in (17) is obtained by:

[TABLE]

Also, the denominator in (17) can be lower bounded as:

[TABLE]

The error bound (17) for the function $\|.\|_{\mathrm{TV},\bm{w}}$ only depends on $\mathcal{D}(\|.\|_{\mathrm{TV},\bm{w}},\bm{x})$ and $\partial\|.\|_{\mathrm{TV},\bm{w}}(\bm{x})$. Further, $\partial\|.\|_{\mathrm{TV},\bm{w}}(\bm{x})$ only depends on $sgn(\bm{d})$ not the magnitudes of $\bm{d}$. So we may choose a vector

[TABLE]

with $sgn(\bm{z})=sgn(\bm{d})$ in (VI-A) to have equality in the last inequality in (VI-A). Therefore, the error in obtaining $\hat{m}_{TV,s,\bm{w}}$ in (149) is at most:

[TABLE]

where in the last step, we used the fact that $|\mathcal{P}_{i}\cap\mathcal{S}_{{g}}|\geq 1,~{}|\mathcal{P}_{i}|\leq\frac{n-1}{L}$ for at least one $i\in[L]$. As a consequence, the error in obtaining $\hat{m}_{\mathrm{TV},\bm{w}}$ is at most $\frac{2\kappa(\bm{\Omega}_{d})}{\sqrt{L(n-1)}}$. ∎

VI-B Optimal Weights

Infimum of (168) gives:

[TABLE]

in which, $\bm{D}=[\bm{1}_{\mathcal{P}_{1}},...,\bm{1}_{\mathcal{P}_{L}}]\in\mathbb{R}^{n-1\times L}$.

In $\mathsf{P}_{\mathrm{TV},\bm{D\omega}}$, we call the weight $\bm{\omega}^{*}=\underset{\bm{\omega}\in\mathbb{R}_{+}^{L}}{\arg\min}~{}~{}\hat{m}_{\mathrm{TV},s,\bm{D\omega}}\in\mathbb{R}_{+}^{L}$ optimal since it asymptotically minimizes the number of measurements required for $\mathsf{P}_{\mathrm{TV},\bm{D\omega}}$ to succeed. Before proving that the optimal weights are in fact unique, we state a beneficial lemma that establish our purpose.

Lemma VI.6*.*

Let $\mathcal{C}:=\partial\|.\|_{\mathrm{TV}}\subseteq\mathbb{R}^{n-1}$. Suppose that $\mathcal{C}$ does not contain the origin. In particular, it is compact and there are upper and lower bounds that satisfy $1\leq\|\bm{z}\|_{2}\leq\sqrt{n-1}$ for all $\bm{z}\in\mathcal{C}$. Also denote the standard normal vector by $\bm{g}\in\mathbb{R}^{n}$. Consider the function

[TABLE]

The function $J$ is strictly convex, continuous at $\bm{\nu}\in\mathbb{R}_{+}^{L}$ and differentiable for $\bm{\nu}\in\mathbb{R}_{++}^{L}$. Moreover, there exist a unique point that minimize $J$.

Proof.

Continuity in bounded points. We must show that sufficiently small changes in $\bm{\nu}$ result in arbitrary small changes in $J(\bm{\nu})$. By the definition of $J_{\bm{g}}(\bm{\nu})$, we have:

[TABLE]

where in the line above, we used the fact that,

[TABLE]

and,

[TABLE]

As a consequence, we obtain:

[TABLE]

Since $\|\bm{\nu}\|_{1}$ is bounded, continuity holds.

Convexity. Let $\bm{\nu}\in\mathbb{R}_{+}^{L}$ , $\tilde{\bm{\nu}}\in\mathbb{R}_{+}^{L}$ and $\theta\in[0,1]$ with $\bm{\upsilon}=\bm{D\nu}$ and $\tilde{\bm{\upsilon}}=\bm{D}\tilde{\bm{\nu}}$. Then we have:

[TABLE]

Since otherwise we have:

[TABLE]

By taking the infimum over $\bm{z},\tilde{\bm{z}}\in\mathcal{C}$, we reach a contradiction. We proceed to prove convexity of $\mathrm{dist}(\bm{g},\bm{\Omega}_{d}^{T}((\bm{D\nu})\odot\mathcal{C}))$.

[TABLE]

Since this holds for any $\epsilon$ and $\tilde{\epsilon}$, $\mathrm{dist}(\bm{g},\bm{\Omega}_{d}^{T}((\bm{D\nu})\odot\mathcal{C}))$ is a convex function. As the square of a non-negative convex function is convex, $J_{\bm{g}}(\bm{\nu})$ is a convex function. At last, the function $J(\bm{\nu})$ is the average of convex functions, hence is convex. In (VI-B), the first inequality comes from the following lemma. In the second inequality, we used the triangle inequality of norms. The third inequality uses the relation (VI-B).

Lemma VI.7*.*

For all $\bm{z}_{1},\bm{z}_{2}\in\mathcal{C}$, the following relation holds.

[TABLE]

Proof of Lemma VI.7.

For $i\in\mathcal{S}_{1}\cup\mathcal{S}_{2}\subseteq\mathbb{R}^{n}$, we have:

[TABLE]

For $i\in\mathcal{S}_{3}$, we can always find a $\bm{z}\in\mathcal{C}$ supported on $\mathcal{S}_{3}$ such that:

[TABLE]

. Since otherwise with $z_{1}(i-1)=z_{2}(i-1)=1$ and $z(i-1)=1$ we reach a contradiction.

For $i\in\mathcal{S}_{4}$, we can always find a $\bm{z}\in\mathcal{C}$ supported on $\mathcal{S}_{4}$ such that:

[TABLE]

Since otherwise with $z_{1}(i)=z_{2}(i)=1$ and $z(i)=1$ we reach a contradiction.

For $i\in\mathcal{S}_{5}$, we can always find a $\bm{z}\in\mathcal{C}$ supported on $\mathcal{S}_{5}$ such that:

[TABLE]

Since otherwise with $z_{1}(i)=z_{2}(i)=1$ and $z(i)=z(i-1)=1$ we reach a contradiction.

For $i\in\mathcal{S}_{g},i-1\notin\mathcal{S}_{g},{\mathcal{\bar{S}}}_{g}$, we have:

[TABLE]

For $i=n$ with $n-1\in{\mathcal{\bar{S}}}_{g}$, we can always find a $z\in\mathcal{C}$ supported on the index $n-1$ such that:

[TABLE]

Since otherwise with $z_{1}(n-1)=z_{2}(n-1)=1$ and $z(n-1)=1$ we reach a contradiction. ∎

Strict convexity. We show strict convexity by contradiction. If $J(\bm{\nu})$ were not strictly convex, there would be vectors $\bm{\nu},\tilde{\bm{\nu}}\in\mathbb{R}_{+}^{L}$ with $\bm{\upsilon}=\bm{D\nu},\tilde{\bm{\upsilon}}=\bm{D}\tilde{\bm{\nu}}$ and $\theta\in(0,1)$ such that,

[TABLE]

For each $\bm{g}$ in (191) the left-hand side is smaller than or equal to the right-hand side. Therefore, in (191), $J_{\bm{g}}(\theta\bm{\nu}+(1-\theta)\tilde{\bm{\nu}})$ and $\theta J_{\bm{g}}(\bm{\nu})+(1-\theta)J_{\bm{g}}(\tilde{\bm{\nu}})$ are almost surely equal(except at a measure zero set) with respect to Gaussian measure. Moreover, we have:

[TABLE]

where, the first inequality comes from (VI.7) and the second inequality stems from the strict convexity of $\|.\|_{2}^{2}$. From (VI.7), it is easy to verify that the set $\bm{\Omega}_{d}^{T}(\bm{\nu}\odot\mathcal{C})$ is a convex set. The distance to a convex set e.g. $\mathcal{E}$ i.e. $\mathrm{dist}(\bm{g},\mathcal{E})$ is a $1$-lipschitz function (i.e. $|\mathrm{dist}(\bm{g},\mathcal{E})-\mathrm{dist}(\tilde{\bm{g}},\mathcal{E})|\leq\|\bm{g}-\tilde{\bm{g}}\|_{2}~{}:~{}\forall~{}\bm{g},\tilde{\bm{g}}\in\mathbb{R}^{n}$) and hence continuous with respect to $\bm{g}$. Therefore, $J_{\bm{g}}(\bm{\nu})$ is continuous with respect to $\bm{g}$. So there exists an open ball around $\bm{g}=\bm{0}\in\mathbb{R}^{n}$ that similar to (VI-B), we may write the following relation for some $\epsilon>0$

[TABLE]

The above statement contradicts with (191) and hence we have strict convexity. Continuity along with convexity of $J$ implies that $J$ is convex on the whole domain $\bm{\nu}\in\mathbb{R}_{+}^{L}$.

Differentiability The function $J_{\bm{g}}(\bm{\nu})$ is continuously differentiable and the gradient for $\bm{\nu}\in\mathbb{R}_{++}^{L}$ is:

[TABLE]

where $\Omega_{d}^{T}(\bm{z}^{*}\odot\bm{D\nu})=\mathcal{P}_{\bm{\Omega}_{d}^{T}(\bm{D\nu}\odot\mathcal{C})}$. Continuity of $\frac{\partial J_{\bm{g}}(\bm{\nu})}{\partial\bm{\nu}}$ at $\bm{\nu}\in\mathbb{R}_{+}^{L}$ stems from the fact that the projection onto a convex set is continuous. For each compact set $\mathcal{I}\subseteq\mathbb{R}_{+}^{L}$ we have:

[TABLE]

where $\nu_{\max}:=\underset{i\in[L]}{\max}~{}\nu(i)$. Therefore, we have:

[TABLE]

where in the last equality, the Lebesgue’s dominated convergence theorem is used.

Attainment of the minimum. Suppose that $\bm{\nu}\geq{\|\bm{g}\|_{2}}\bm{1}_{L\times 1}$ where $\bm{g}\in\mathbb{R}^{n}$ is a fixed vector. With this assumption we may write:

[TABLE]

where, $\nu_{\min}:=\underset{i\in[L]}{\min}~{}\nu(i)$. By squaring (VI-B), we reach:

[TABLE]

Using the relation $\mathds{E}\|\bm{g}\|_{2}\geq\frac{n}{\sqrt{n+1}}$ and Marcov’s inequality we obtain:

[TABLE]

Then we reach:

[TABLE]

where in (VI-B), the first inequality stems from total probability theorem, the second inequality comes from (136). From (VI-B), we find out that $J(\bm{\nu})>J(\bm{0})$ when $\nu>(2^{\frac{1}{4}}+1)\|\bm{\Omega}_{d}^{\dagger}\|_{2\rightarrow 2}{\sqrt{p}}\bm{1}_{L\times 1}$. Therefore, the unique minimizer of $J$ must occur in the interval $[\bm{0},(2^{\frac{1}{4}}+1)\|\bm{\Omega}_{d}^{\dagger}\|_{2\rightarrow 2}{\sqrt{p}}\bm{1}_{L\times 1}]$ ∎

Proposition VI.8*.*

Let $\bm{x}$ be a non-uniform $s$-gradient sparse vector in $\mathbb{R}^{n}$ with parameters $\{\rho_{i}\}_{i=1}^{L}$ and $\{\alpha_{i},\alpha_{i}^{\prime},\beta_{i},\beta_{i}^{\prime},\gamma_{i},\gamma_{i}^{\prime},\varsigma_{i},\varsigma_{i}^{\prime},\xi_{i},\breve{\xi}_{i}\}_{i=1}^{L}$. Then there exist unique optimal weights $\bm{\omega}^{*}\in\mathbb{R}_{+}^{L}$ (up to a positive scaling) that minimize $\hat{m}_{\mathrm{TV},s,\bm{D\omega}}$. Moreover, the optimal weights $\bm{\omega}^{*}\in\mathbb{R}$ are obtained by simultaneously solving the following integral equations.

[TABLE]

where in (VI.8),

[TABLE]

Proof.

Define $\mathcal{C}:=\partial\|.\|_{\mathrm{TV}}(\bm{x})$ and use Lemma VI.2 to obtain:

[TABLE]

where, $\Psi_{t,\bm{D\omega}}(\bm{\alpha},\bm{\alpha}^{\prime},\bm{\beta},\bm{\beta}^{\prime},\bm{\gamma},\bm{\gamma}^{\prime},\bm{\varsigma},\bm{\varsigma}^{\prime},\bm{\xi},\breve{\bm{\xi}})$ is defined in (149). Also, we used a change of variable $\bm{\nu}=t\bm{\omega}$ to convert multivariate optimization problem to a single variable optimization problem. Thus, the function $J_{\mathrm{TV}}(\bm{\nu})$ is obtained via the following equation:

[TABLE]

By considering Lemma VI.6 and $\bm{D}:=[\bm{1}_{\mathcal{P}_{1}},...,\bm{1}_{\mathcal{P}_{L}}]_{n-1\times L}$, the function $J_{\mathrm{TV}}(\bm{\nu})$ is continuous and strictly convex and thus the unique minimizer can be obtained using $\nabla J_{\mathrm{TV}}(\bm{\nu})=\bm{0}\in\mathbb{R}^{L}$ which leads to (VI.8). ∎

Theorem VI.9*.*

Let $\bm{x}\in\mathbb{R}^{n}$ be a non-uniform $s$-gradient sparse vector with parameters $\{\rho_{i}\}_{i=1}^{L}$ and $\{\alpha_{i},\alpha_{i}^{\prime},\beta_{i},\beta_{i}^{\prime},\gamma_{i},\gamma_{i}^{\prime},\varsigma_{i},\varsigma_{i}^{\prime}\}_{i=1}^{L}$. Then, the number of measurements required for $\mathsf{P}_{\mathrm{TV},\bm{D}\bm{\omega}^{*}}$ is upper bounded by the sum of number of measurements required for $\mathsf{P}_{\mathrm{TV}}$ to recover each $\{\bm{x}_{\mathcal{P}_{i}}\in\mathbb{R}^{n}\}_{i=1}^{L}$ by considering the variations between partitions and lower bounded by the whole number of measurements one needs to recover each $\{\bm{x}_{\mathcal{P}_{i}}\in\mathbb{R}^{n}\}_{i=1}^{L}$ separately ignoring the variations between partitions. Written in a mathematical form, we have:

[TABLE]

where,

[TABLE]

Proof.

As previously defined in (149), with optimal weights, the upper bound for normalized number of measurements required for $\mathsf{P}_{\mathrm{TV},\bm{D}\bm{\omega}^{*}}$ to succeed is given by:

[TABLE]

On the other hand, by using the relation $\underset{\mathcal{A}}{\inf}(f+g)\geq\underset{\mathcal{A}}{\inf}(f)+\underset{\mathcal{A}}{\inf}(g)~{}:~{}\forall f,g:\mathcal{A}\rightarrow\mathbb{R}$, we have:

[TABLE]

As a consequence, regarding the error bounds obtained in Propositions VI.4 and VI.5, we reach (204). ∎

VII Simulations

In this section, we numerically verify our theoretical bounds on the optimal weights and number of measurements required for successful recovery in the three inherent structures. Remark that, MATLAB package cvx [28] is used to implement the optimization problems.

VII-A Entrywise sparsity in a redundant dictionary

In this subsection, we intend to find how the required number of Gaussian linear measurements for successful recovery of $\mathsf{P}_{1,\Omega}$, scales with the signal sparsity $s$ in the redundant discrete cosine transform (DCT) dictionary $\Omega\in\mathbb{R}^{p\times n}$. For this purpose, we consider a grid of $(m,s)$ values and count number of successful recovery of $\mathsf{P}_{1,\Omega}$ in $100$ trials to find the empirical probability. The heatmap in Figure 1 shows the empirical probability that is consistent with the theory obtained by (21). In a separate experiment, we generate a random vector $\bm{x}\in\mathbb{R}^{n}$ that is $10$-sparse in the DFT domain $\bm{\Omega}\in\mathbb{R}^{100\times 90}$ and consider two sets $\mathcal{P}_{1}$ and $\mathcal{P}_{2}$ in the DFT domain $\bm{\Omega}\in\mathbb{R}^{100\times 90}$ that partition $[100]$ with $\alpha_{1}=\frac{7}{10}$ and $\alpha_{2}=\frac{3}{90}$. In each trial, we recover the signal from Gaussian i.i.d linear measurements by solving $\mathsf{P}_{1,\bm{\Omega}}$ and $\mathsf{P}_{1,\bm{\Omega},\bm{D}\bm{\omega}^{*}}$. The optimal weights $\bm{\omega}^{*}\in\mathbb{R}^{2}$ are obtained by solving the equation (76) using MATLAB function fzero. Figure 2 shows that $\mathsf{P}_{1,\bm{\Omega},\bm{D}\bm{\omega}^{*}}$ with optimal weights needs less measurements than $\mathsf{P}_{1,\bm{\Omega}}$.

VII-B Block sparsity

In this subsection, we simulate how the required number of measurements for successful recovery of $\mathsf{P}_{1,2}$ scales with block sparsity. The heatmap in Figure 3 shows the empirical probability of success which is consistent with the theory obtained by (V.1). In the second experiment, we generate an $s=50$-block sparse random vector $\bm{x}\in\mathbb{R}^{1280}$ with $128$ blocks of equal size $10$. Then, we consider three sets $\mathcal{P}_{1}$, $\mathcal{P}_{2}$ and $\mathcal{P}_{3}$ with $\alpha_{1}=\frac{27}{50},~{}\alpha_{2}=\frac{18}{20},~{}\alpha_{3}=\frac{5}{58}$ that partition the set of blocks $[128]$. We implement $100$ trials. In each trials, we solve $\mathsf{P}_{1,2,\bm{\omega}^{*}}$ with optimal weights $\bm{\omega}^{*}$ and recover $\bm{x}\in\mathbb{R}^{1280}$ from $m$ Gaussian linear measurements. The optimal weights are obtained via the equation (138) by MATLAB function fzero. Figure 4 shows that $\mathsf{P}_{1,2,\bm{D}\bm{\omega}^{*}}$ with optimal weights needs less measurements than $\mathsf{P}_{1,2}$ for recovering $\bm{x}\in\mathbb{R}^{1280}$.

VII-C Gradient sparsity

In this subsection, we investigate the probability that $\mathsf{P}_{\mathrm{TV}}$ and $\mathsf{P}_{\mathrm{TV},\bm{D}\bm{\omega}^{*}}$ successfully recover an $s$-gradient sparse vector from i.i.d Gaussian linear measurements. We consider two partitions $\mathcal{P}_{1}$ and $\mathcal{P}_{2}$ with known probability of intersection with consecutive and individual supports. In the first experiment, we consider a strict nonuniform case with random partition. In this case, we fixed the accuracy of $\mathcal{P}_{1}$ and $\mathcal{P}_{2}$ at $\frac{|\mathcal{P}_{1}\cap\mathcal{S}_{g}|}{|\mathcal{P}_{1}|}=\frac{5}{6}$ and $\frac{|\mathcal{P}_{2}\cap\mathcal{S}_{g}|}{|\mathcal{P}_{2}|}=\frac{1}{13}$, respectively. The set $\mathcal{P}_{1}$ is most probable to include the elements of the gradient support $\mathcal{S}_{g}$ than the set $\mathcal{P}_{2}$. We calculate the optimal weights $\bm{\omega}^{*}$ by solving the relation (VI.8) using MATLAB function fsolve for each measurement and trial. We repeat the experiment $20$ times. Figure 5 indicates the empirical probability computed over $20$ trials. As it is clear from Figure 5, $\mathsf{P}_{\mathrm{TV},\bm{D}\bm{\omega}^{*}}$ needs less measurement than $\mathsf{P}_{\mathrm{TV}}$ to reach the same success rate. In the second experiment, we consider a two partitioned $4$-gradient sparse vector $\bm{x}\in\mathbb{R}^{21}$ with random accuracy and partition size 888The smooth signal may be uniform or non-uniform.. We repeat the experiment $50$ times and calculate the optimal weights $\bm{\omega}^{*}$ via (VI.8). Figure 6 indicates the empirical probability computed over $50$ trials.

VIII Conclusion

In this paper, we investigated structured signal recovery with extra prior information for three popular class of low-complexity models: entrywise, block and gradient sparse vectors. We found explicit formulas for number of Gaussian linear measurements needed to recover an structured vector. An important question central to the weighted convex minimization problems is the optimal choices of weights. In this paper, we provide a general framework based on techniques developed in conic integral geometry to find optimal weights. Also, we proved that the optimal weights are unique up to a positive scaling in the aforementioned low-complexity models. Additionally, we have included numerical experiments that verify out theoretical results.

Appendix A Proof of Proposition IV.2

Proof.

In this appendix, we prove Proposition IV.2. In the following, we relate $\mathcal{D}(\|\bm{\Omega}.\|_{1},\bm{x})$ to $\mathcal{D}(\|.\|_{1},\bm{\Omega x})$.

[TABLE]

where in the above equation, we used the facts that $\mathcal{D}(\|\bm{\Omega}.\|_{1},\bm{x})$ is a closed convex set and $\bm{\Omega}^{\dagger}\bm{\Omega}=\bm{I}$ for redundant dictionary $\bm{\Omega}\in\mathbb{R}^{p\times n}$ with $p\geq n$. In the following, we state Sudakov-Fernique inequality which helps to control the superimum of a random variable by that of a simpler random variable and is used to find an upper bound for $\omega(\mathcal{D}(\|\bm{\Omega}.\|_{1},\bm{x})\cap\mathds{B}_{1}^{n})$.

Theorem A.1*.*

(Sudakov-Fernique inequality). Let $T$ be a set and $\mathbf{X}=(X_{t})_{t\in T}$ and $\mathbf{Y}=(Y_{t})_{t\in T}$ be Gaussian processes satisfying $\mathds{E}[X_{t}]=\mathds{E}[Y_{t}]~{}:~{}\forall t\in T$ and $\mathds{E}|X_{t}-X_{s}|^{2}\leq\mathds{E}|Y_{t}-Y_{s}|^{2}~{}:~{}\forall s,t\in T$, then

[TABLE]

[TABLE]

where in (A), $\bm{h}\in\mathbb{R}^{p}$ is a standard normal vector with i.i.d components. The first inequality in (A) comes from the relation (A), the second inequality comes from Theorem A.1 with $X_{\bm{t}}=\langle\bm{g},\bm{\Omega}^{\dagger}\bm{t}\rangle$ and $Y_{\bm{t}}=\|\bm{\Omega}^{\dagger}\|_{2\rightarrow 2}\langle\bm{h},\bm{t}\rangle$ and the fact that:

[TABLE]

In the last inequality of (A), we used the change of variable $\bm{w}=\|\bm{\Omega}\|_{2\rightarrow 2}^{-1}\bm{z}$. The following Proposition states the connection between statistical dimension and Gaussian width. In the same setting, the work of Amelunxen et al. [17, Proposition 10.2] relates the notion of statistical dimension to Gaussian width with a different proof.

Proposition A.2*.*

Let $\mathcal{C}\subseteq\mathbb{R}^{n}$ be a convex cone. Then the following relation between statistical dimension and Gaussian width holds.

[TABLE]

Proof.

[TABLE]

where the first inequality comes from Gaussian poincaré inequality [29, Theorem 1.6.4] for $1$-lipschitz function $\|\mathcal{P}_{\mathcal{C}}(\bm{g})\|_{2}$, the forth equality stems from convexity of the problem $\underset{{\bm{z}\in\mathcal{C}^{\circ}}}{\inf}\|\bm{g}-\bm{z}\|_{2}$ and strong duality. On the other hand, we have:

[TABLE]

where the first inequality uses the Jensen’s inequality for concave functions. ∎

By using Proposition A.2 and (A), we have:

[TABLE]

∎

Appendix B Proof of Theorem IV.1

Proof.

The proof is based on [17, Theorem 6.1] with a bit different terminology. By taking $\lambda=\sqrt{8\log\frac{4}{\eta}}\sqrt{n}$ when $m\geq\delta(\mathcal{D}(\|\bm{\Omega}.\|_{1},\bm{x}))+\mu+\lambda$ we have:

[TABLE]

where in the above equation, $h_{i}(\mathcal{D}(\|\bm{\Omega}.\|_{1},\bm{x}))$ and $t_{i}(\mathcal{D}(\|\bm{\Omega}.\|_{1},\bm{x}))$ are the $i$th half-tail functional and $i$th tail-functional, respectively which are defined in [17, Definition 5.8].

The relation of the tolerance $\eta$ in Theorem III.3 and $\eta^{\prime}$ is given below:

[TABLE]

∎