Stability Analysis for a Class of Sparse Optimization Problems
Jialiang Xu, Yun-Bin Zhao
TL;DR
This paper establishes a stability result for $ ext{l}_1$-minimization in sparse optimization, generalizing previous results by introducing a new property of sensing matrices, which enhances understanding of signal recovery stability.
Contribution
The paper introduces the restricted weak range space property (RSP) of sensing matrices, generalizing previous concepts, and establishes a stability result for $ ext{l}_1$-minimization in a broad class of $ ext{l}_0$-minimization problems.
Findings
Introduces the restricted weak RSP of sensing matrices.
Establishes a generalized stability theorem for $ ext{l}_1$-minimization.
Includes several existing stability results as special cases.
Abstract
The sparse optimization problems arise in many areas of science and engineering, such as compressed sensing, image processing, statistical and machine learning. The -minimization problem is one of such optimization problems, which is typically used to deal with signal recovery. The -minimization method is one of the plausible approaches for solving the -minimization problems, and thus the stability of such a numerical method is vital for signal recovery. In this paper, we establish a stability result for the -minimization problems associated with a general class of -minimization problems. To this goal, we introduce the concept of restricted weak range space property (RSP) of a transposed sensing matrix, which is a generalized version of the weak RSP of the transposed sensing matrix introduced in [Zhao et al., Math. Oper. Res., 44(2019),…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSparse and Compressive Sensing Techniques · Microwave Imaging and Scattering Analysis · Numerical methods in inverse problems
Stability Analysis for a Class of Sparse Optimization Problems
\nameJialiang Xua and Yun-Bin Zhaob Yun-Bin Zhao. Email: [email protected] a,b School of Mathematics, University of Birmingham, Edgbaston, Birmingham, B15 2TT, United Kingdom
Abstract
The sparse optimization problems arise in many areas of science and engineering, such as compressed sensing, image processing, statistical and machine learning. The -minimization problem is one of such optimization problems, which is typically used to deal with signal recovery. The -minimization method is one of the plausible approaches for solving the -minimization problems, and thus the stability of such a numerical method is vital for signal recovery. In this paper, we establish a stability result for the -minimization problems associated with a general class of -minimization problems. To this goal, we introduce the concept of restricted weak range space property (RSP) of a transposed sensing matrix, which is a generalized version of the weak RSP of the transposed sensing matrix introduced in [Zhao et al., Math. Oper. Res., 44(2019), 175-193]. The stability result established in this paper includes several existing ones as special cases.
keywords:
Sparsity optimization; -minimization; stability; optimality condition; Hoffman theorem; restricted weak range space property.
1 Introduction
The sparsity is a useful assumption under which the sparse optimization models arise frequently in many areas in science and engineering. Let , and be three given full-row-rank matrices. Let and be given vectors and be a positive number. Consider the following sparse optimization model:
[TABLE]
where is called the ‘-norm’ which counts the number of nonzero components of , and and are given nonnegative parameters satisfying . Many problems in signal and image processing (see, e.g., [6, 13, 17]) and statistical regressions [23] can be formulated as the form (1) or its special cases. In problem (1), the constraint is motivated by some practical applications. For instance, many signal recovery models might need to include certain constraints reflecting special structures of the target signal. For simplicity, we define
[TABLE]
and write the problem (1) as
[TABLE]
The following -minimization models are clearly the special cases of (1):
[TABLE]
The problem (C1) is often called the standard -minimization problem [17, 8, 28]. Two structured sparsity models, called the nonnegative sparsity model [8, 7, 17, 28] and the monotonic sparsity model (isotonic regression) [24, 23], are also the special cases of the model (1).
It is well known that -minimization is a useful method to solve the -minimization problem. By replacing the -norm with the -norm in problem (1), we immediately obtain the -minimization problem
[TABLE]
Similar to its counterpart, the problem (2) includes the following special cases:
[TABLE]
The problem (D2) is often called quadratically constrained basis pursuit [17, 10, 28], and it reduces to (D1) if , which is called standard -minimization or the basis pursuit [12, 8, 19, 26, 17]. The problem (D4) is the type of Dantzig Selectors [9, 17].
From both numerical and theoretical viewpoints, it is important to know how close the solutions of - and -minimization problems are. To address this question, one needs to study the stability of -minimization methods. The stability of a sparse optimization method can be described as follows: For any in the feasible set of a sparse optimization problem, the solution generated by the method satisfies the following bound:
[TABLE]
where and are constants, and is called the error of the best -term approximation of the vector (see, e.g., [12, 17]):
[TABLE]
In this paper, we establish a stability result for the -minimization method (2). The stability of (D1) and (D2) has been investigated by Donoho, Candès, Tao, Romberg and others [14, 13, 6, 7, 8, 12, 25, 16, 3] under various assumptions such as the so-called restricted isometry property (RIP) of order , mutual coherence, stable null space property (NSP) of order or robust NSP of order . The RIP of order was introduced by Candès and Tao [8] to study the stability of -minimization. The singular-value-property-based stability analysis for (D1), (D2) and the Dantzig Selector have also been performed by Tang and Nehorai in [22].
A new and unified stability analysis for -minimization methods has been developed by Zhao, Jiang and Luo [29] under the assumption of weak RSP of order , which has been proven as a necessary and sufficient condition for the standard -minimization to be stable. The main differnece between the weak-RSP-based-analysis and existing ones lies in the constants and in (3). Specifically, the constants and in (3) are determined by the RIP or NSP constant in existing analysis [17, 3, 8]. However, in [29, 28], these constants are determined by the so-called Robinson’s constant. Motivated by the new analysis tool introduced in [29], we develop the stability result for the model (2) in this paper under the assumption of restricted weak range space property () of order (which will be introduced in next section). Our result extends the stability theorem for -minimization established by Zhao et al. [29, 30, 28].
This paper is organized as follows. In Section 2, we introduce the concept of restricted weak of order . An approximation of the solution set of (2) will be discussed in Section 3. Then, in Section 4, we show the main stability result of this paper. Finally, some special cases are discussed in Section 5.
Notation
The field of real numbers is denoted by and the -dimensional Euclidean space is denoted by . Let and be the sets of nonnegative and nonpositive vectors, respectively. Unless otherwise stated, the identity matrix of suitable size is denoted by . Given a vector , , and denote the vectors with components , and , , respectively. The cardinality of the set is denoted by and the complementary set of is denoted by , i.e., . For a given vector , denotes the vector supported on . denotes the entry of the matrix in row and column . For the set , denotes the submatrix of obtained by deleting the columns indexed by . For a matrix , represents the absolute version of , i.e., . is the range space of . , where , is a norm, called the -norm of . is called the -norm of . For , is the matrix norm induced by - and -norms.
2 Restricted weak range space property
The of order of a transposed matrix was first introduced in [26, 27] to develop a necessary and sufficient condition for the uniform recovery of sparse signals via -minimization. Zhao et al. [29] generalised the of order to the following weak of order to develop a stability theory for convex optimization algorithms:
Definition 2.1** (weak of order ).**
Given a matrix , is said to satisfy the weak order if for any two disjoint sets satisfying , there exists a vector such that
[TABLE]
In [29, 28], it was shown that the weak of order is a sufficient condition for the stability of many convex optimization methods, and it is also a necessary stability condition for many optimization methods.
Different from the problems (D1)-(D4), the problem (2) is more general than these models. To investigate the stability of the problem (2), we need to extend the notion of weak RSP of order to the so-called restricted weak of order , which is defined as follows:
Definition 2.2** (Restricted weak of order ).**
Given matrices and , the pair is said to satisfy the restricted weak of order if for any two disjoint sets satisfying , there exists a vector such that \eta=\left(A^{T},B^{T}\right)\left(\begin{array}[]{c}\nu\\ h\end{array}\right) where , and
[TABLE]
It is worth mentioning that a generalized version of the RSP of order is also used in [31] to study the exact sign recovery in 1-bit compressive sensing.
3 Approximation of (2) and its solution set
By introducing the slack variables , , and , the problem (2) can be rewritten as
[TABLE]
where is the vector of ones in and is the unit -ball defined as . The unit ball can be also described as
[TABLE]
Denote the set by
[TABLE]
and hence the solution set of (4) can be represented as
[TABLE]
where is the optimal value of (4). By replacing in (6) with a polytope , we can get the relaxation of , denoted by , i.e.,
[TABLE]
The polytope can approximate to any level of accuracy provided that is chosen suitably. Recall the Hausdorff metric of two sets :
[TABLE]
Following the analysis in [29, 28] (see Lemmas 5.1, 5.2 and 5.3 in [29]), we can obtain the following lemma:
Lemma 3.1**.**
Let be the given number in problem (2). Then for any , there exists a polytope approximation of satisfying and
[TABLE]
In the remainder of this paper, we fix and choose the polytope such that and satisfy (8). The polytope can be represented as the intersection of a finite number of half spaces:
[TABLE]
where are some unit vectors (i.e., ), and is an integer number. By adding the half spaces
[TABLE]
to , where is the th column of the identity matrix, we obtain the following polytope:
[TABLE]
We define as the collection of the vectors and in , that is,
[TABLE]
Clearly, still satisfies (8) in Lemma 3.1, i.e.,
[TABLE]
In the remainder of the chapter, we use the above defined polytope Let and let be the matrix with column vectors in . Thus can be written as
[TABLE]
where is the vector of ones in .
By replacing by , we obtain the following approximation of the optimal value of (2):
[TABLE]
The associated approximation problem of (2) can be written as
[TABLE]
The solution set of (10) is
[TABLE]
Note that implies that . So we can see that . By the definition of , we also have . In the next section, we prove the main result for the problem (2).
4 Main result
Introducing a variable yields the following equivalent form of (10):
[TABLE]
The solution set of (12) is given as (11). Note that the above optimization problem is equivalent to a linear programming problem. In fact, the constraint can be rewritten as where is the vector of ones in . Thus the model (12) can be rewritten explicitly as the linear programming problem
[TABLE]
The dual problem of (13) is given as follows:
[TABLE]
The optimality condition yields the following lemma:
Lemma 4.1**.**
Denote by . Then is an optimal solution of (10) if and only if there exists a vector , where is the set given as
[TABLE]
Clearly, holds for every . The set can be written as the form
[TABLE]
where the vectors and
[TABLE]
The matrices and in (15) are given as follows:
[TABLE]
where the matrices , , , and and are given as follows:
[TABLE]
[TABLE]
[TABLE]
In the above matrices, [math]’s are zero matrices with suitable sizes and , and are the , and identity matrices, respectively.
To prove the main stability result, we also need the next two Lemmas.
Lemma 4.2** (Hoffman [18, 21]).**
Let and be two given matrices and the set be given as
[TABLE]
For any vector , there exists a vector satisfying
[TABLE]
where is a constant determined by and .
The constant is also called the Robinson constant. We also use the following lemma in the proof of the main result in this section.
Lemma 4.3** ([28, 30]).**
Let be the projection of into the convex set , i.e., Let the three convex compact sets , and satisfy that and Then for any and any the following holds:
[TABLE]
We also define two types of constants. Let
[TABLE]
be a matrix with full row rank. Given three positive numbers , we define the constants and as follows:
[TABLE]
We will use the above constants together with the specific constants and in the stability analysis of (2). The main result is given as follows.
Theorem 4.4**.**
Let the problem data of (2) be given, and the matrix be given in (18) with full row rank. Let be the polytope given in (9) satisfying (8). If satisfies the restricted weak of order , then for any , there is an optimal solution of (2) satisfying the bound
[TABLE]
where is the Robinson constant determined by in (16), and are the constants given in (19a) and (19b), and are five given positive numbers (allowing to be ) satisfying
[TABLE]
In particular, if is a feasible solution of (2), then there is an optimal solution of (2) such that
[TABLE]
Proof.
Let be any given vector in and be the fixed polytope given in (9) satisfying (8) in Lemma 3.1. We let satisfy that
[TABLE]
With such a choice of , we have
[TABLE]
Let be the support set of largest absolute entries of , and and be the sets such that
[TABLE]
Clearly, . Let be the complementary set of . Clearly, , and are disjoint. Under the assumption of restricted weak RSP of order , there exists a vector such that for some and satisfying
[TABLE]
Now we construct a feasible solution to the dual problem (14).
Constructing (). Set and as follows:
[TABLE]
Such and satisfy that
[TABLE]
Constructing (–). Note that is a matrix with full row rank. There must exist an invertible matrix of , denoted by , where with . Denote the complementary set of by Then we construct a vector satisfying and which imply that
[TABLE]
Let () be the vector obtained by keeping the positive (negative) components of and setting the remaining components to [math]. By using the vector , – can be constructed as follows:
[TABLE]
which implies that
[TABLE]
Constructing . Without loss of generality, we suppose that the first columns in are and are the second columns of . The components of can be assigned as follows:
[TABLE]
From this choice of , we can see that
[TABLE]
Constructing . Let . Such a choice of together with the choice of – implies that
[TABLE]
Constructing . Let . Clearly, due to .
With the above choice of , we deduce from (26), (29), (30) and (31) that
[TABLE]
Let and be defined as follows:
[TABLE]
For the vector where is constructed above, by Lemma 4.2, there exists a vector where is given in Lemma 4.1 and written as (15), such that
[TABLE]
where is the Robinson constant determined by given by (16). Since the vector satisfies (24) and (32), the inequality (33) can be simplified to
[TABLE]
In the reminder of the proof, we estimate the terms on the right-hand side of (34). Note that the vectors in are unit vectors. It is easy to see that
[TABLE]
The value of in (23) implies that Therefore we have
[TABLE]
Due to (27), (29) and (30), we have
[TABLE]
The fact (due to the restricted weak RSP of order ) and the triangle inequality imply that
[TABLE]
Now we deal with the right-hand side of the above inequality. First, by using the index sets and , we have
[TABLE]
It follows from and (25) that
[TABLE]
Then we obtain
[TABLE]
By using the restricted weak of order , we have
[TABLE]
where and is defined in (19b). Moreover, we have
[TABLE]
Recall that is determined in (19a). Then . Similarly, can be obtained. Due to , we have
[TABLE]
Let be three given positive numbers and be two given numbers satisfying (21). For the term in (36), it follows from Hölder inequalities that
[TABLE]
Let be given as (19a), i.e.,
[TABLE]
Thus we have
[TABLE]
Similarly, the following inequalities holds
[TABLE]
Due to (37), (38), (40) and (41), the inequality (36) is reduced to
[TABLE]
where .
Note that It follows from (34), (35) and (42) that
[TABLE]
We recall the three sets , and , where and are the solution sets of (2) and (10), given as (6) and (11), respectively, and is given as (7) with . Clearly, . Let denote the projection of onto , that is,
[TABLE]
Note that the three sets are compact convex sets satisfying and . Then by applying Lemma 4.3 with , and , we have
[TABLE]
Since satisfies (8), it implies that
[TABLE]
Let Combination of the above inequality and (43) yields the desired results (20). If is the feasible solution of (2), then and
[TABLE]
and thus the desired error bound (22) is also obtained. ∎
Based on Theorem 4.4, the error bound for the solutions of (1) and (2) can be stated as follows.
Corollary 4.5**.**
For any optimal solution of (1), there is an optimal solution of (2) estimating with the error:
[TABLE]
where the constants , , , and are given as in Theorem 4.4.
5 Special cases
Firstly, by setting different values of and , the problem can reduce to several special cases, and the corresponding stability results for these special cases can be obtained from (20) and (22) immediately. Note that if any of and is zero, the constant in (20) and (22) will be simplified as well. For example, if , the constant is reduced to . The following table shows the form of the constant for different choices of and .
Note that for any case with , we have so that where and . Thus instead of using Lemma 4.3, the stability results can be immediately obtained from (43).
Secondly, without matrix , the problem (2) is reduced to
[TABLE]
In this case, the restricted weak of order is reduced to the standard weak of order , which means . In fact, the upper bound of in (39) can be improved to
[TABLE]
Then in order to obtain a tighter bound, can be replaced by
[TABLE]
Thus we have . Similarly, the constants and are replaced by and , respectively. Clearly, in this case, . Let . Then the bound (22) is reduced to
[TABLE]
Similarly, we list the constants for different choices of in the following table.
Note that when , we have due to the fact . Moreover, in this case, setting yields
[TABLE]
which is the bound for the following -minimization established by Zhao and Li [30] (see also in Zhao [28]):
[TABLE]
Last but not least, our analysis can also apply to 1-bit basis pursuit [31], which can be viewed as a special case of our model (2). The stability result for the 1-bit basis pursuit in [31] can be obtained immediately from Theorem 4.4 by setting .
6 Conclusion
In this paper, we have studied the stability issue of the -minimization method (2). To establish our results, we introduced the restricted weak RSP of order which is a mild assumption governing the stability of sparsity-seeking algorithms. Under this assumption, we use the classic Hoffman theorem and Lemma 4.3 to show that the -minimization method (2) is stable and thus the error between the solutions of the problems (1) and (2) can be measured in terms of the best term approximation and the problem data (see Theorem 4.4). The result developed in this paper can apply to a range of problems with constraints defined by -, -, and -norms.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] J. Andersson and J. O. Str o ¨ ¨ o \ddot{\mathrm{o}} mberg, On the theorem of uinform recoery of structured random matrices , IEEE Trans. Inform. Theory 60 (2014), pp. 1700–1710.
- 2[2] P. T. Boufounos, Greedy sparse signal reconstruction from sign measurements , Porc. 43rd Asilomar Conf. Signals, Systems and Computers, 2009, pp. 1305–1309.
- 3[3] J. Cahill, X. Chen, and R. Wang, The gap between the null space property and restricted isometry property , Linear Algebra Appl. 501 (2016), pp. 363–375.
- 4[4] T. Cai, L. Wang, and G. Xu, New bounds for restricted isometry constants , IEEE Transactions on Information Theory. 56 (2010), pp. 4388–4394.
- 5[5] T. Cai and A. Zhang, Sharp RIP bound for sparse signal and low-rank matrix recovery , Applied and Computational Harmonic Analysis. 35 (2013), pp. 74–93.
- 6[6] E. Candès, Compressive sampling , Proceedings of the International Congress of Mathematicians. 3 (2006), pp. 1433–1452.
- 7[7] E. Candès, J. Romberg and T. Tao, Stable signal recovery from incomplete and inaccurate measurements , Communications on Pure and Applied Mathematics 59 (2006), pp. 1207–1223.
- 8[8] E. Candès and T. Tao, Decoding by linear programming , IEEE Transactions on Information Theory 51 (2005), pp. 4203–4215.