Sparse Optimization Problem with s-difference Regularization
Yuli Sun, Xiang Tan, Xiao Li, Lin Lei, Gangyao Kuang
TL;DR
This paper introduces an s-difference regularization for sparse recovery, providing theoretical equivalence with L0 constraints, efficient solution methods, and demonstrating superior performance through numerical experiments.
Contribution
It proposes a novel s-difference regularization, establishes its theoretical properties, and develops an efficient FBS algorithm with convergence guarantees.
Findings
The s-difference regularization effectively promotes sparsity.
The FBS method converges to a stationary point.
Numerical results show improved recovery performance.
Abstract
In this paper, a s-difference type regularization for sparse recovery problem is proposed, which is the difference of the normal penalty function R(x) and its corresponding struncated function R (xs). First, we show the equivalent conditions between the L0 constrained problem and the unconstrained s-difference penalty regularized problem. Next, we choose the forward-backward splitting (FBS) method to solve the nonconvex regularizes function and further derive some closed-form solutions for the proximal mapping of the s-difference regularization with some commonly used R(x), which makes the FBS easy and fast. We also show that any cluster point of the sequence generated by the proposed algorithm converges to a stationary point. Numerical experiments demonstrate the efficiency of the proposed s-difference regularization in comparison with some other existing penalty functions.
| Function type | |||||
|---|---|---|---|---|---|
| Convex, Separable | |||||
| Convex, non-Separable | |||||
| Non-convex, Separable |
|
||||
|
|
|||||
| Non-convex, Non-separable | |||||
| -ADMM | -DCA |
|
AIHT |
|
|
|
|||||||||||||||||
| 256 | 1024 | 48 |
|
|
|
|
|
|
|
||||||||||||||
| 512 | 2048 | 96 |
|
|
|
|
|
|
|
||||||||||||||
| 768 | 3072 | 144 |
|
|
|
|
|
|
|
||||||||||||||
| 1024 | 4096 | 192 |
|
|
|
|
|
|
|
||||||||||||||
| 1280 | 5120 | 240 |
|
|
|
|
|
|
|
||||||||||||||
| 1536 | 6144 | 288 |
|
|
|
|
|
|
|
||||||||||||||
| 1792 | 7168 | 336 |
|
|
|
|
|
|
|
||||||||||||||
| 2018 | 8192 | 384 |
|
|
|
|
|
|
|
| -ADMM | -DCA |
|
AIHT |
|
|
|
|||||||||||||||||
| 256 | 1024 | 48 |
|
|
|
|
|
|
|
||||||||||||||
| 512 | 2048 | 96 |
|
|
|
|
|
|
|
||||||||||||||
| 768 | 3072 | 144 |
|
|
|
|
|
|
|
||||||||||||||
| 1024 | 4096 | 192 |
|
|
|
|
|
|
|
||||||||||||||
| 1280 | 5120 | 240 |
|
|
|
|
|
|
|
||||||||||||||
| 1536 | 6144 | 288 |
|
|
|
|
|
|
|
||||||||||||||
| 1792 | 7168 | 336 |
|
|
|
|
|
|
|
||||||||||||||
| 2018 | 8192 | 384 |
|
|
|
|
|
|
|
| -ADMM | -DCA |
|
AIHT |
|
|
|
|||||||||||||||||
| 256 | 1024 | 48 |
|
|
|
|
|
|
|
||||||||||||||
| 512 | 2048 | 96 |
|
|
|
|
|
|
|
||||||||||||||
| 768 | 3072 | 144 |
|
|
|
|
|
|
|
||||||||||||||
| 1024 | 4096 | 192 |
|
|
|
|
|
|
|
| -ADMM | -DCA |
|
AIHT |
|
|
|
|||||||||||||||||
| 256 | 1024 | 48 |
|
|
|
|
|
|
|
||||||||||||||
| 512 | 2048 | 96 |
|
|
|
|
|
|
|
||||||||||||||
| 768 | 3072 | 144 |
|
|
|
|
|
|
|
||||||||||||||
| 1024 | 4096 | 192 |
|
|
|
|
|
|
|
| Methods |
|
|
|
|
||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Rel.Err | Iter/Time | Rel.Err | Iter/Time | Rel.Err | Iter/Time | Rel.Err | Iter/Time | |||||||||
| -ADMM | 1.098E-04 | 1.357E-04 | 1.198E-01 | 7.485E-02 | ||||||||||||
| (DCA-ADMM) | 2.298E-05 | 178/0.05 | 2.501E-05 | 170/0.05 | 7.182E-02 | 302/0.08 | 4.430E-02 | 511/0.12 | ||||||||
| (PDCA) | 3.735E-05 | 530/0.13 | 4.063E-05 | 460/0.12 | 1.179E-01 | 5120/1.46 | 1.005E-01 | 3559/1.08 | ||||||||
| (FBS) | 1.368E-05 | 126/0.04 | 3.059E-06 | 65/0.03 | 6.190E-02 | 195/0.06 | 4.264E-02 | 108/0.05 | ||||||||
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSparse and Compressive Sensing Techniques ยท Photoacoustic and Ultrasonic Imaging ยท Microwave Imaging and Scattering Analysis
Sparse Optimization Problem with -difference Regularization
Yuliย Sun, Xiangย Tan, Xiaoย Li, Linย Lei, Gangyaoย Kuangโ College of Electronic science and Technology, National University of Defense Technology, Changsha 410073, China
Abstract
In this paper, a -difference type regularization for sparse recovery problem is proposed, which is the difference of the normal penalty function and its corresponding -truncated function . First, we show the equivalent conditions between the constrained problem and the unconstrained -difference penalty regularized problem. Next, we choose the forward-backward splitting (FBS) method to solve the non-convex regularizes function and further derive some closed-form solutions for the proximal mapping of the -difference regularization with some commonly used , which makes the FBS easy and fast. We also show that any cluster point of the sequence generated by the proposed algorithm converges to a stationary point. Numerical experiments demonstrate the efficiency of the proposed -difference regularization in comparison with some other existing penalty functions.
Index Terms:
Sparse constrained, Forward-Backward Splitting, proximal operator, difference of convex, truncated function.
I Introduction
I-A Background
In recent years, sparse optimization problems have drawn lots of attention in many applications such as compressive sensing, machine learning, image processing and medical imaging. Signal and image processing problems are usually expressed as
[TABLE]
where is the linear or non-linear operator, is the observation data, and represents the observation noise or error. Since problem (1) is often ill-posed and the error is unknown, solving (1) is difficulty. To overcome this ill-posed problem, we need to make some constraints to narrow the solution space, such as the prior sparsity of the signals. Then the problem can be formulated as
[TABLE]
where the loss function is the data fidelity term related to (1), for example, the least square (LS) loss function or the least-absolute (LA) loss function ; is the regularizes function to penalize the sparsity of . Intuitively, should be selected as the -norm , represents the number of nonzero elements in . However, minimizing the -norm is equivalent to finding the sparsest solution, which is known to be NP-hard problem. A favorite and popular approach is using the -norm convex approximation, i.e., to replace the [1]. This model has been widely used in many different applications, such as radar systems [2-3], communications [4], computed tomography (CT) [5] and magnetic resonant imaging (MRI) [6]. It has been proved that the signal can be recovered by this model under some assumption of the operator , such as the restricted isometry property (RIP) of when the operator is a sensing matrix [1]. However, the -norm regularization tends to underestimate high-amplitude components of as it penalizes the amplitude uniformly, unlike -norm in which all nonzero entries have equal contributions. This may lead to reconstruction failures with the least measurements [7-8], and brings undesirable blocky images [9-10]. It is quite well-known that the when it promotes sparsity, the -norm does not provide a performance close to that of the -norm, and lots of theoretical and experimental results in CS and low-rank matrix recovery suggest that better approximations of the -normand matrix rank give rise to better performances.
Recently, researchers began to investigate various non-convex regularizes to replace the -norm regularization and gain some better reconstructions. In particular, the (quasi)-norm with [11-16], can be regarded as a interpolation between the and , and a continuation strategy to approximate the as . The optimization strategies include half thresholding [14, 17-20] and iterative reweighting [11-12, 15]. Other non-convex regularizations and algorithms have also been designed to outperform -norm regularization and seek better reconstruction: capped -norm [21-23], transformed -norm [24-26], sorted -norm [27-28], the difference of the and -norms () [29-31], the log-sum penalty (LSP) [8], smoothly clipped absolute deviation (SCAD) [32-33], minimax-concave penalty (MCP) [34-36].
On the other hand, there are some approaches which do not approximate the -norm, such as the iterative hard thresholding (IHT) algorithm [37-38], which operate directly on the regularized cost function or the -sparse constrained optimization problem. Moreover, there are some acceleration methods for the IHT: accelerated IHT (AIHT) [39], proximal IHT (PIHT) [40], extrapolated proximal IHT (EPIHT) [41] and accelerated proximal IHT [42]. Meanwhile, there are some researchers transformed the -norm problem into an equivalent difference of two convex functions, and then using the difference of convex algorithm (DCA) and the proximal gradient technique to solve the subproblem [43-44].
To address these nonconvex regularization problems, many iterative algorithms are investigated by researchers, such as the DCA [45-48] (or Convex-ConCave Procedure (CCCP) [49], or the Multi-Stage (MS) convex relaxation [22]), and its accelerate versions: Boosted Difference of Convex function Algorithms (BDCA) [50] and proximal Difference-of-Convex Algorithm with extrapolation (pDCAe) [51], the alternating direction method of multipliers (ADMM) [52], split Bregman iteration (SBI) [53], General Iterative Shrinkage and Thresholding (GIST) [54], nonmonotone accelerated proximal gradient (nmAPG) [55], which is an extension of the APG [56].
I-B Contributions
In many applications, the non-convex -norm based regularization has its advantages over the convex -norm , such as image restoration [41, 53, 57-58], bioluminescence [59], CT [9-10], MRI reconstruction [60-61]. Thus, in this paper, we are interested in the following constrained problem
[TABLE]
i.e. this -sparse problem tries to find the solution minimizing under the constraint that the number of non-zero coefficients below a certain value, where .
This paper can be viewed as a natural complement and extension of Gotoh et al. framework [43]. First, we rewrite the constrained problem (3) as difference of two functions, one of which is the convex or nonconvex function and the other is the corresponding truncated function . Then, we consider the unconstrained minimization problem by using this -difference type regularizations. Second, we propose fast approaches to deal with this non-convex regularizes function, which is based on a proximal operator corresponding to . Moreover, we derive some cheap closed-form solutions for the proximal mapping of with some commonly used , such as , , , LSP, MCP and so on. Third, we prove the convergence performance of the proposed algorithm, and show that any cluster point of the sequence generated by the proposed algorithm converges to a stationary point. We also show a link between the proposed algorithm with some related regularizations and algorithms. Finally, we evaluate the effectiveness of the proposed algorithm via numerical experiments. The reconstruction results demonstrate that the proposed difference penalty function with closed-form solutions is more accurate than the -norm and other non-convex regularization based methods, and faster than the DCA based algorithms.
I-C Outline and notation
The rest of this paper is structured as follows. In section 2, we define the constrained sparse optimization. In section 3, we propose the reconstruction algorithm by using the proximal operator with closed-form solutions. In section 4, we provide some theorems to demonstrate the convergence of the proposed algorithm. Section 5 presents the numerical results. In the end, we provide our conclusion in section 6.
Here, we define our notation. For a vector , it can be written as , and its -norm is defined as {\left\|{\bf{x}}\right\|_{p}}={\left({\sum\nolimits_{n}{{{\left|{{x_{n}}}\right|}^{p}}}}\right)^{{\raise 1.50694pt\hbox{\scriptstyle 1}\kern-0.70004pt/\kern-1.04996pt\lower 0.75346pt\hbox{\scriptstyle p}}}}. Especially, the -norm of is defined as . Given a matrix , the transpose of is denoted by , the maximum eigenvalue of is defined as . Some of the arguments in this paper use sub-vectors. The letters , denote sets of indices that enumerate the elements in the vector . By using this sets as subscripts, represents the vector that setting all elements of to zero except those in the set . The iteration count is given in square bracket, e.g., . denotes the inner product, represents the sign of a quantity with . We also use the notation , and if the function is defined as the composition , then we write .
Given a proper closed function , the subgradient of at is given by
[TABLE]
In addition, if is continuously differentiable, then the subdifferential reduces to the gradient of denoted by .
II Penalty representation for -sparse problem
Inspired by Gotoh et al. work of [43], in which they expressed the -norm constraint as a difference of convex (DC) function:
[TABLE]
where and , which named top- norm, denotes the sum of top-elements in absolute value. This notation is also known as the largest- norm (or called CVaR norm in [62-63]). Precisely,
[TABLE]
where denotes the element whose absolute value is the -th largest among the elements of vector , i.e., . For convenience of description, we define the set , then we have . By using as the set difference, we have .
In this work, we consider a more general -difference function instead of to replace the -norm constraint, where can be convex or nonconvex, separable or non-separable, and is the best term approximation to , that is, any -sparse vectors that minimize . By using the definition of , we have
[TABLE]
Let , , we defined a class of penalty functions as follows (without loss of generality, functions and mentioned thought this paper all satisfy Property 1 ).
Property 1**.**
The penalty functions satisfy the following properties.
(a)
(b)
(c) is a continuous function which can be written as the difference of two convex (DC) functions, that is, , where and are convex functions.
Proposition 1**.**
The penalty functions listed on Table 1 all satisfy Property 1.
See appendix A for the Proof of Proposition 1.
Remark 1. For the separable , and is continuous, symmetrical and strictly increasing on , if is convex, then satisfies Property 1; if is nonconvex, while it can be written as the difference of two convex functions as , then also satisfies Property 1.
It is easy to see that the penalty function in Ref. [43] is a special case of .
With the Property 1(b), we consider the following unconstrained minimization problem associated with (3):
[TABLE]
where is the penalty parameter. We make the following assumptions on the above formulation thought the paper, which are standard in image processing and many CS field.
Assumption 1**.**
* is continuously differentiable with Lipschitz continuous gradient, i.e., there exists such that*
[TABLE]
Assumption 2**.**
* is bounded from below.*
From (8), we can find that the difference between penalty and other penalty function, such as , , and MCP, is that there is no punishment in model (8) when the sparsity level of is under , since is equal to zero as . Meanwhile, the selection of the weighting parameter has importance influence on the performance of the reconstruction. On the one hand, should be big enough to give a heavy cost for constraint violation: . On the other hand, if is too big, the reconstruction is mostly over regularized. In light of this, we need the adjust the value of iteratively based on the convergence speed. The next Theorem ensures that problem (8) is equivalent to the original -sparse constraint problem (3) as we take the limit of , which can be proved in a similar manner to Theorem 17.1 in [71].
Theorem 1**.**
Let be an increasing sequence with and suppose that is an optimal solution of (8) with . Then, any limit point of is also optimal to (3).
See Appendix B for the proof.
In addition to Theorem 1, we have some stricter conclusions for the parameter under some assumptions of and .
Proposition 2**.**
If is Lipschitz continuous with constant , i.e., , and is an optimal solution of (8) with some . Suppose that there exists a constant such that for any . Then if , is also optimal to (3).
See Appendix C for the proof.
Remark 2. Suppose that is -Lipschitz continuous and the regularization is . Then if , any optimal solution of (8) is also optimal to (3).
Remark 3. Suppose that is -Lipschitz continuous. If we choose as , then any optimal solution of (8) is also optimal to (3) when . This can be proved by using that
[TABLE]
If we choose , then the condition of is that . Meanwhile, we can obtain similar conclusions for the which are the difference of and MCP, or SCAD functions.
The next proposition, which is similar to Theorem 3 in [43], but with wider scope and stricter conclusion, shows another exact penalty parameters requirement for with Lipschitz continuous gradient .
Proposition 3**.**
If Assumption 1 is satisfied and is an optimal solution of (8) with some . Suppose that there exists a constant such that for any , and there exists a constant such that for any , Then if , is also optimal to (3).
See Appendix D for the proof.
Remark 4. Suppose that and . If we choose as , () and (), then any optimal solution of (8) is also optimal to (3) when , and , respectively.
Remark 5. Similarly to Theorem 4 in [43] by replacing penalty function with ordinary function , we have the following conclusions without proof. If the conditions in Proposition 3 are satisfied, and suppose that , where is symmetric and , then is also optimal to (3) if , where denotes the unit vector in the -th coordinate direction.
III Forward-Backward Splitting for the regularization of difference of two functions
In this section, we use the FBS to solve the unconstrained minimization (8). Moreover, we derive closed-form solutions for the proximal mapping of some special regularization -difference , and this makes FBS more efficient.
III-A Forward-Backward Splitting and proximal operator
Each iteration of forward-backward splitting applies the gradient descent of followed by a proximal operator. That is
[TABLE]
where is the step size, and the FBS is sometimes called the proximal gradient (PG) algorithm. The proximal operator is defined as
[TABLE]
with parameter .
The equation (11) can be broken up into a forward gradient step using the function , and a backward step using the function . The proximal operator plays a central role in the analysis and solution of optimization problems. For example, the soft shrinkage operator, which is a proximal operator for -norm regularizer, has been widely used in CS and rendering many efficient algorithms. The proximal operator also has been successfully used with some nonconvex regularizers, such as , SCAD, LSP [64], and MCP [52, 65]. Usually, the closed-form solution of the proximal operator needs some special properties on , such as convexity or separability (e.g., the -norm, LSP, MCP, and other various separable functions in [66]), Next, we will focus on the solution of (12) with separable and non-separable -difference .
III-B Closed-form solution of the proximal operator
Denote as
[TABLE]
Let be the optimal solution of (12), i.e., , then we have the following Proposition.
Proposition 4**.**
* if and only if .*
Proof:
Necessary condition: note that for any , and when , we have . Thus if , the optimal solution is . Sufficient condition: assume by contradiction that , then we select an arbitrary non-zero dimension in , and construct as {\tilde{x}_{i}}=\left\{{\begin{array}[]{*{20}{c}}{0,}&{i\neq j}\\ {{{\rm{y}}_{j}},}&{i=j}\end{array}}\right.. Then we have
[TABLE]
This contradicts the optimality of . Thus if , must be equal to zero. โ
Proposition 5**.**
For , if , then we have . If , then we have .
Proof:
We prove it by establishing contradiction. If there exits any when , then we select an arbitrary one and we construct as {\tilde{x}_{j}}=\left\{{\begin{array}[]{*{20}{c}}{{x}_{j}^{*},}&{j\neq i}\\ {-{x}_{j}^{*},}&{j=i}\end{array}}\right.. We have
[TABLE]
The inequality follows from that has the opposite sign as and . Since we have not changed the absolute value of and , then we have . Combing this and (15), we have . This contradicts the optimality of and proves that when . On the other hand, we can prove that when by using a similar method. This completes the proof. โ
Next, we focus on the closed-form solutions of with different types of .
Proposition 6**.**
If is separable, i.e., and each is strictly increasing on , we have
[TABLE]
where denotes the identity operator, and is the index of the -th largest amplitude of , i.e., .
See Appendix E for the proof.
Remark 6. Note that if in (16). Suppose that there exits one or more components of , having the same amplitude of , i.e., , . Then there exits solutions of as there are arrangements of .
Remark 7. If , then the solution of (12) is
[TABLE]
where denotes the soft shrinkage operator given by
[TABLE]
Remark 8. If , then the solution of (12) is
[TABLE]
Remark 9. If is the MCP (A.3), that is {r_{i}}\left({{x_{i}}}\right)=\left\{{\begin{array}[]{*{20}{c}}{\left|{{x_{i}}}\right|-{{x_{i}^{2}}\mathord{\left/{\vphantom{{x_{i}^{2}}{\left({2\theta}\right)}}}\right.\kern-1.2pt}{\left({2\theta}\right)}},}&{\left|{{x_{i}}}\right|\leq\theta}\\ {{\theta\mathord{\left/{\vphantom{\theta{2,}}}\right.\kern-1.2pt}{2,}}}&{\left|{{x_{i}}}\right|>\theta}\end{array}}\right. (),then the solution is: under the condition of , if or , then ; otherwise . When , if or , then ; otherwise . If is the LSP (A.2), that is , then the solution is: if , then ; otherwise , and , where is a set composed of 3 elements or 1 element. If , then
[TABLE]
where and . Otherwise, .
Proposition 6 gives the solution of the (12) under the conditions of with separable and strictly increasing properties. In fact, there are some other commonly used separable and non-convex also have the closed-form solution similar as (16), such as with [14], however, these does not satisfy the Property 1(c), so they are not within the scope of this article. Next, we consider two special non-separable cases as the reference for other non-separable regularizations.
Proposition 7**.**
If , then the solution of (12) is that: when ,
[TABLE]
when ,
[TABLE]
See Appendix F for the proof.
Proposition 8**.**
If , , then the solution of (12) is that:
1) When , for ,
[TABLE]
for ,
[TABLE]
where for , and for .
2) When , if , , , and suppose that there are components of having the same amplitude of , i.e., . is an optimal solution of (12) if and only if it satisfies , , and when . In this case, there are infinite many solutions, equations (A.40) and (A.41) are two solution examples. When , and any of these conditions , , cannot be satisfied, the solution is
[TABLE]
3) When , the solution is the same as (25).
We apply the similar proof framework in Ref. [29] for the fast minimization. See Appendix G for the proof.
Remark 10. When , then reduces to , and the corresponding solution of (23, 24, 25) reduces to (17) as in Remark 7.
IV Convergence analysis
The purpose of this section is to demonstrate that the sequence of obtained from the FBS for (8) is convergent.
Theorem 2**.**
If Assumption 1 and 2 are satisfied and , let be the sequence generated by the FBS for (8), the following statements hold.
1) The sequence is bounded.
2) .
3) Any accumulation points of is a stationary point of .
Proof:
- Rewrite (8) and consider the following inequality
[TABLE]
where the in the third equation is the expression (13) with replaced by and set . The first inequality comes from Assumption 1, and the second inequality is based on the fact that is the optimal solution of the . When , we have for all . Due to the level-boundedness of (Assumption 2), therefore the sequence is bounded.
- Summing both sides of (26) from to , we can obtain
[TABLE]
Since , we can deduce that from the above relation obviously.
- Since the sequence is bounded, there exists a subsequence of , denoted as , converging to an accumulation point . Considering that minimizer is a critical point of (13) and , we have
[TABLE]
Let , by using from the above conclusion and considering the semi-continuity of , and , we have that . Therefore, is a critical point of problem (8). This completes the proof. โ
From the proof of Theorem 2, we have that is a necessary optimality condition of the FBS. Therefore, we can use as a quantity to measure the convergence performance of the sequence to a critical point .
Theorem 3**.**
If , let be the sequence generated by the FBS for (8), then for every , we have
[TABLE]
Proof:
Summing the inequality (26) over , we can obtain
[TABLE]
When , we have is monotonically decreasing, which means that . Substitute this into (30), we have
[TABLE]
This completes the proof. โ
In fact, we may have a stricter conclusion for the convergence speed as can be smaller than in (26).
Proposition 9**.**
If is separable, i.e., ๏ผ and each is strictly increasing on , then we have
[TABLE]
where , , and .
See Appendix H for the proof.
From Proposition 9, we can find that if is the same as .
V Extensions
In this section, we discuss some related algorithms for solving (8), show a link between the DC function with other regularization functions, and simply extend to rank-constrained problem.
V-A Related algorithms
Here, we discuss some related algorithms. When is convex, it is an intuitive idea that using the DCA to solve the minimization (8). Since can be written as the DC functions, i.e.,, the objective function can be naturally decomposed into
[TABLE]
The corresponding DCA solves the minimization problem as
[TABLE]
where . Although this problem is convex, it does not necessarily have closed-form solution and the computational cost is very expensive for large-scale problems.
On the other hand, since is continuously differentiable with -Lipschitz continuous gradient, we can use the Sequential Convex Programming (SCP) [67] to solve problem (8) by updating as
[TABLE]
Meanwhile, the SCP can be thought as a variant of DCA with DC decomposition:
[TABLE]
The subproblem can be written as
[TABLE]
Due to that the subproblem (37) can be solved by using the proximal operator, Ref. [43] and [44] call this type DCA as proximal DCA (PDCA). For some simple form , subproblem (37) also has closed-form solution. For example, and . In the numerical experiment, we will compare the FBS with this PDCA and show that the FBS is more efficient than PDCA in this problem. Meanwhile, as is a DC function, the FBS reduces to the GIST algorithm proposed in [54].
To improve the performance of the FBS, some acceleration methods can be used in the proximal framework. Such as the Nonmonotone Accelerated proximal gradient (nmAPG) method [55], the extrapolation method in PDCA (pDCAe) [51] and the backtracking line search initialized method with Barzilai-Borwein (BB) rule [68] in GIST [54].
V-B Comparing with other regularization
From the previous discussion, we have illustrated that the DC function can replace the -norm constraint. And in Theorem 1 and Proposition 2, we have proved that the unconstrained problem (8) is equal to the original sparsity constrained problem (3) if we select proper parameter . On the other hand, in the minimization problem (8), can also be considered as a regularizes function. Then, we can investigate its performance from the aspect of sparsity metric. Figure 1 shows the contours of various regularizers.
From Fig.1, we can find that the level curves of approach the and axes as the values get small, hence promoting sparsity. Inspire by Sidky et al. work of [69] and Rahimi et al. work of [70], where they using toy examples to illustrate the advantages of and , respectively, we also use a similar example to show that with some special data sets , the tends to select a sparser solution.
Example 1 Let and define
{\bf{A}}:=\left[{\begin{array}[]{*{20}{c}}1&{-1}&0&0&0&0\\ 0&1&{-1}&0&0&0\\ 0&1&2&1&0&0\\ 2&1&1&0&1&0\\ {0.5}&{0.5}&3&0&0&{-1}\end{array}}\right], {\bf{b}}:=\left[{\begin{array}[]{*{20}{c}}0\\ 0\\ {15}\\ {20}\\ {40}\end{array}}\right]
It is straightforward that any general solutions of have the form of for a scalar . The sparest solution occurs at for the sparsity of being 3, and some local solutions include for sparsity being 4 and for sparsity being 5. We plot the various regularize function with respect to , including , (), , , MCP () of (A.3) and the proposed with , , , , MCP, and .
From Fig. 2, we can find that all these regularized functions are not differentiable at the values of , and , where the corresponding sparsity of are all small than 6. However, only the and the -difference can find the sparsest vector at as a global minimum, where the other functions find as the minimum and lead to the sparsity of being 4.
V-C Extend to rank-constrained problem
Similar as in [43], the penalty function can also be extended to rank-constrained problem based on the connection between the -norm on and the rank function for a matrix. The rank-constrained minimization problem can be formulated as
[TABLE]
where is a non-negative integer with . As the rank of a matrix is equal to the number of its nonzero singular values, i.e., , where represents the singular value vector of and is the -th largest term, then we can construct the penalty functions , that satisfies Property 1 (b) and (c), where for and for else. Replace the rank constraint with the DC penalty function and consider the unconstrained problem:
[TABLE]
Then we can use the FBS, DCA or ADMM algorithms to solve this rank-constrained problem.
VI Numerical experiments
In this section, simulations are performed to demonstrate the proposed conclusions and evaluate the performance of the -difference regularization. We mainly apply four methods in comparison with the proposed algorithm: (1) the -norm regularization based -ADMM [72], (2) the -norm () regularization based half thresholding [14], (3) the -norm regularization based accelerate IHT (AIHT) [39], (4) the difference of the and -norms () regularization based -DCA [31]. We choose the representative as for comparing. All experiments are performed in MATLAB 2015b running on ASUS laptop with Intel (R) Core (TM) i7-8550U CPU, 8 GB of RAM and 64-bit Windows 10 operating system.
We focus on the following least squares problem:
[TABLE]
and conduct experiments on simulated vector signals.
We test two types of matrices : the random Gaussian matrix with i.i.d. standard Gaussian entries and being normalized that each column has unit norm, and the random partial DCT matrix which is formed by randomly select rows from the full DCT matrix. For the original sparse vector , we generate it with random index set and draw non-zero elements with standard normal distribution. The observation is , where is zeros for the noiseless test, and Gaussian noise for the contaminated measurements. The initial value for all the methods is an approximated solution of the minimization using ADMM after iterations. The max iteration for all these methods is except for DCA, whose max internal iteration is and the max external iteration is . The stopping condition is set to be .
In the first study, we look at the success rates with 100 random instances under the noise-free condition, in which we set the size of matrices as . Here we consider a recovery as successful if the relative error of recovery (Rel.Err) satisfies . In addition, we set sparsity parameter to the ground truth for the proposed -difference . Fig 3 plots the success rates of the comparing methods for both the Gaussian matrix and the partial DCT matrix. From this, we can find that the -difference regularization with has the best performance for both Gaussian matrix and partial DCT matrix, the is comparable to -DCA, followed by and half thresholding, which outperform the -ADMM.
In the second study, we focus on the recovery quantity of these methods under different sizes of matrix. For the noiseless case , we set for FBS and for the ADMM and other types methods, set , and consider for . Here we also set the sparsity threshold parameter to for the AIHT and -difference . For each triple , we generate 30 random realizations. Table 2 and 3 list the mean and standard deviation of Rel.Err for Gaussian matrix and partial DCT matrix, respectively. We also test these methods in presence of Gaussian noise as . We set for FBS and for the ADMM and other types methods, and consider for . The recovery performance is listed in Table 4 and 5 for comparing. From Table 2 to 5, we can find that the -difference with the ground truth sparsity threshold parameter can provide a quite competitive or slightly superior performance when comparing with AIHT and other methods under the noise-free conditions. However, under the condition of noise, AIHT performance decreases rapidly, while the -difference still able to provide a relatively best result.
In the third study, we focus on the accuracy and efficiency of the methods under fixed matrix and the sparsity level as . To illustrate the benefit of the closed-form solutions of proposed -difference regularization, we selectively analysis the performance of DCA, PDCA and FBS under the condition of the same regularization . The DCA solves the minimization problem (40) by using (34), that is
[TABLE]
where . This problem can be solved by ADMM as
[TABLE]
We denote this method as DCA-ADMM for short. The PDCA solve the minimization problem (40) by using (37), that is
[TABLE]
and it can be solved by using soft shrinkage operator (18). We denote this method as PDCA for short. The FBS solve the problem by using closed-form solution (17) in Remark 7.
Figure 4 shows the convergence performance of three methods under noise-free condition with partial DCT matrix, which is measured by the Log-Rel.Err (defined as ) versus iteration numbers. Table 6 lists the mean of relative error, iteration number and computational time (in seconds) under the noise-free and Gaussian noise conditions as . From Figure 4 and Table 6, it is clear that the FBS with closed-form method leads to less error and converges faster than the DCA type methods.
From the definition of -difference , it is easy to understand that the parameter plays an important role in the algorithm. Here we focus on the problem of how the select the proper with the fixed matrix and as . Figure 5 shows the performance of -difference under different from 1 to 1000. In addition to use the FBS with closed-form solution as Proposition 8, we also consider the approximate DCA-ADMM using the similar solution of (42) but set . This method is not a true DCA due to that the decomposition is not the convex function, however, this DCA-ADMM still works well as shown in Figure 5. From Figure 5, we can find that once the parameter s is less than the true sparsity , the performance of FBS with closed-form will drop sharply, however, the DCA-ADMM almost unaffected. This is probably because that the FBS solve the problem as the hard thresholding way when is smaller than in Proposition 8, whereas the DCA-ADMM make full use of the nonconvex and bring better results than the -norm methods. According to this deduction, designing an adaptive penalty parameter for FBS is quite necessary, which also is our future work. The good performance of DCA-ADMM also shows the superiority of this -difference regularization from another angle.
From Figure 5, we also have a suggestion that if we already have a preliminary range of judgements about sparsity based on prior knowledge, i.e., , then we suggest that decreases from the , but no less than , or just set be equal to when the range of sparsity is not very large. Here, we also introduce an adjustment strategy to estimate the parameter when we donโt know the prior sparsity range: set , where constant is given. Some experiments show that this adjustment strategy usually can find the approximate true sparsity level , which means that it maybe can be used to estimate the sparsity of the unknown signal.
VII Conclusion
In this paper, we propose a new -difference type penalty function for the sparse optimization problem, which is the difference of the normal convex or nonconvex penalty function and its corresponding -truncated function. To solve this nonconvex regularization problem, we select the FBS method based on the proximal operator, which have some cheap closed-form solutions for commonly used , such as , , and so on. The convergence and effectiveness of the proposed algorithm are proved and demonstrated by the theoretical proof and numerical experiments, respectively. In addition, we observed that the DCA with -difference regularization gives better recovery results than the FBS using close-form solutions when the parameter is less than the true sparsity, which motivate us to find an adaptive strategy for the penalty and sparsity parameters in the future.
Appendix A Proof of Proposition 1
To prove the Proposition 1, we use the following Lemma:
Lemma 1**.**
If is convex, then for any , is also convex.
Proof:
let , since is convex, then is convex. Then the can be written as a pointwise maximum of convex functions:
[TABLE]
Then we have that is convex.
-
For the convex and separable , such as and , and the convex and non-separable functions , such as , it is obviously that they fulfilling (a) and (b). Then by using Lemma 1, it completes the Property 1(c).
-
For the non-convex and separable functions , where are equations (A.2), (A.3) and (A.4) corresponding to LSP, MCP and SCAD, respectively.
[TABLE]
[TABLE]
[TABLE]
Property 1(a) and (b) is obvious. Then we need give the DC formulations for . Take the LSP for example, we have that
[TABLE]
which means that is convex as is convex. Then we can rewrite as
[TABLE]
where and are two convex functions. For MCP and SCAD, we can obtain similar formulations in the same way.
- For the non-convex and non-separable functions, when , we have . When , it is easy to see that . When , we have ; otherwise , then , then we have , which means that , and this is contradiction to . Meanwhile, can be formulated as
[TABLE]
when is the non-separable LSP, denoted as , Property 1(a) and (b) are obvious. Note that can be thought as a composition function , where and , by using the above deduction, we have that is convex. Then can be rewritten as
[TABLE]
For the non-separable type MCP and SCAD, we can obtain similar formulations in the same way. โ
Appendix B Proof of Theorem 1
Proof:
This theorem can be proved in a similar manner to Theorem 17.1 in [71]. Let be an optimal solution of (3), that is,
[TABLE]
Since minimizes (8) at , we have that
[TABLE]
By rearranging this expression, we have
[TABLE]
Suppose that is a limit point of , then there exits an infinite subsequence such that . By taking the limit as , , on both side of (A.11)
[TABLE]
Therefore, we have that , which means that is feasible to (3). Moreover, by taking the limit as for on (A.10), we have that
[TABLE]
Since is feasible to (3) and is an optimal solution of (3), then is also optimal to (3). โ
Appendix C Proof of Proposition 2
Proof:
For simplicity, we use instead of for an optimal solution of (8) with some . First, we proof that . If , which means that . We construct a vector as , easily we have that . When , we have
[TABLE]
This contradicts the optimality of . Then we have that satisfies the -sparse constraint of (3). Let be an optimal solution of (3), then we have
[TABLE]
The inequality comes from that is the optimal solution of (8). This means that is also optimal to (3). โ
Appendix D Proof of Proposition 3
Proof:
Similar to the previous proof of Proposition 2, we use instead of for an optimal solution of (8) with some . Assume by contradiction that , which means that . By constructing , we have
[TABLE]
The first inequality using Assumption 1 that
[TABLE]
The third inequality follows from that
[TABLE]
(A.16)) contradicts the optimality of , then we have that satisfies the -sparse constraint of (3). Then we can prove that is also optimal to (3) similar as the previous proof of Proposition 2. โ
Appendix E Proof of Proposition 6
Proof:
Suppose that is the optimal solution of (12). First, we prove that if .we have ; otherwise , then we construct as and . Whether or or , we always have that and . As , then we can obtain . However, this contradicts the optimality of .
Next, we prove that . To prove this, we need to prove that for all . We can do this one by one, i.e., we look at first. Easily, we have ; otherwise we construct , as is strictly increasing on and symmetrical, thus we have the contradiction , then we have . By using this deduction, we can prove that in a similar way. At last, we have .
Rewrite as
[TABLE]
As , we have that and . This completes the proof. โ
Appendix F Proof of Proposition 7
Proof:
First, we show that when , we also have if .we have . Otherwise, we can always construct a , which swap the absolute value of and as the same way in the proof of Proposition 6, then we can obtain a smaller objective value. As proved in Proposition 4, if and only if . Then, we only need to consider the case .
- If , then we have
[TABLE]
Easily, we have that if , then and .
When , the first-order optimality condition optimality condition of minimizing is that
[TABLE]
By using Proposition 5, we have that in (A.21). Using (A.21), we have
[TABLE]
Substitute one equation of (A.22) into another, we have
[TABLE]
By using the equation , we have
[TABLE]
[TABLE]
[TABLE]
Substitute these into (A.21), then we have
[TABLE]
- If , then we have a similar conclusion as Remark 6.
From the above deduction, we have the expression of in (21) and (22) when . This completes the proof. โ
Appendix G Proof of Proposition 8
Proof:
Similar to the previous proof of Proposition 6, we have that
[TABLE]
As proved in Proposition 4, if and only if . Then, we just consider the condition of . Firstly, we suppose that , then we have .
The first-order optimality condition of minimizing is that
[TABLE]
[TABLE]
where is a subgradient.
- First case, when . Easily we have by using Proposition 5: . When , then , so we have ; when , then , and we also have .
For , if , then . Otherwise, for this , if , then based on Proposition 5. As , the left-hand side (LHS) of (A.30) is positive, while the right-hand side (RHS) of (A.30) nonpositive; if , then based on Proposition 5. The LHS of (A.30) is negative, while the RHS of (A.30) is nonnegative; if , we have based on (A.28).
For , if any , then we have based on (A.30). For this , we construct a vector as
[TABLE]
For , we have , then we can obtain
[TABLE]
For , we have
[TABLE]
Substitute (A.32) into (A.33), we have
[TABLE]
By using the equation , we have
[TABLE]
[TABLE]
[TABLE]
Substitute these into (A.29) and (A.30), then we have: for ,
[TABLE]
for ,
[TABLE]
- If , for , suppose that there are components of having the same amplitude of , i.e., .
For , we have . Otherwise, for this , if , then based on Proposition 5. Easily, we have and from (A.30). Meanwhile, as , we have , then , and this contradicts to the equation in (A.30). If , then based on Proposition 5, we have and from (A.30). However, as , this also contradicts to the equation . If , we have based on (A.28). Then we obtain that for .
For , if there exits , for this we have , then we obtain and . Substitute this into (A.29), we have . As , then we have that there exits if and only if the conditions of , , and are all satisfied. In this case, there are infinite many solutions, and all these should satisfy , and when . For example,
[TABLE]
or
[TABLE]
When any of these conditions , , cannot be satisfied, we have for . Then we have . Substitute this into (A.29), we have for . Then the solution can be expressed as
[TABLE]
- If , for , suppose that there are components of having the same amplitude with , i.e., .
For , we have . Otherwise, for this , as , we have based on (A.28). Then we obtain from (A.30), and we have , which means that through (A.30). Since , then we can obtain based on Proposition 5. If , then we have and obtain , which means that . However, this contradicts . If , we have , then we can obtain , which contradicts . Then, we obtain that for .
For , if there exits , then we have as the sign of and are opposite. For , from (A.29), we have
[TABLE]
If , we have , which contradicts . So, when , we have , and then the solution is
[TABLE]
If , for , suppose there are components of and . From (A.30), we have . Substitute this into from (A.29), we have
[TABLE]
Reconsider the expression of , and using the first-order optimality condition, we have
[TABLE]
Then we have , and we need to find the with the largest norm among all that satisfying (A.29) and (A.30). From this, we have that should be zero to make the largest in (A.45). So, when , we have the solution the same as (A.44).
- If , for , we have . Otherwise, we can construct a vector , which is equal to except setting these corresponding to be zero. Then we can obtain a smaller objective value, which contradicts the optimality of . For , we have . Then the solution can be expressed as (A.44).
Once again, if there exits one or more components of , having the same amplitude of , then we have a similar conclusion as Remark 6. This completes the proof.
โ
Appendix H Proof of Proposition 9
Proof:
Let be the optimal solution of (13) with , then we have
[TABLE]
The third equation comes from Proposition 6, and the last inequality is based on the property of subgradient. Then we have
[TABLE]
where , , and . Substitute this into (26) then we have (32). This completes the proof. โ
Acknowledgment
This work is partially supported by the National Natural Science Foundation of China (61701508).
References
[1] E.J. Candรจs, J. Romberg, T. Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory, 52(2): 489โ509, 2006.
[2] Patel V M, Easley G R, Healy Jr D M, et al. Compressed synthetic aperture radar[J]. IEEE Journal of selected topics in signal processing, 2010, 4(2): 244-254.
[3] Yang J, Thompson J, Huang X, et al. Random-frequency SAR imaging based on compressed sensing[J]. IEEE Transactions on Geoscience and Remote Sensing, 2013, 51(2): 983-994.
[4] Berger C R, Wang Z, Huang J, et al. Application of compressive sensing to sparse channel estimation[J]. IEEE Communications Magazine, 2010, 48(11): 164-174.
[5] Chen Z, Jin X, Li L, et al. A limited-angle CT reconstruction method based on anisotropic TV minimization[J]. Physics in Medicine Biology, 2013, 58(7): 2119.
[6] Lustig, Michael, David Donoho, and John M. Pauly. โSparse MRI: The application of compressed sensing for rapid MR imaging.โ Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 58.6 (2007): 1182-1195.
[7] Chartrand R, Staneva V. Restricted isometry properties and nonconvex compressive sensing[J]. Inverse Problems, 2008, 24(3): 035020.
[8] Candes E J, Wakin M B, Boyd S P. Enhancing sparsity by reweighted โ 1 minimization[J]. Journal of Fourier analysis and applications, 2008, 14(5-6): 877-905.
[9] Sun Y, Tao J. Few views image reconstruction using alternating direction method viaโnorm minimization[J]. International Journal of Imaging Systems and Technology, 2014, 24(3): 215-223.
[10] Yu-Li S, Jin-Xu T. Image reconstruction from few views by โ0-norm optimization[J]. Chinese Physics B, 2014, 23(7): 078703.
[11] Chartrand R. Exact reconstruction of sparse signals via nonconvex minimization[J]. IEEE Signal Processing Letters, 2007, 14(10): 707-710.
[12] Chartrand R, Yin W. Iteratively reweighted algorithms for compressive sensing[C]//2008 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2008: 3869-3872.
[13] Krishnan D, Fergus R. Fast image deconvolution using hyper-Laplacian priors[C]//Advances in neural information processing systems. 2009: 1033-1041.
[14] Xu Z, Chang X, Xu F, et al. regularization: A thresholding representation theory and a fast solver[J]. IEEE Transactions on neural networks and learning systems, 2012, 23(7): 1013-1027.
[15] Lai M J, Xu Y, Yin W. Improved iteratively reweighted least squares for unconstrained smoothed minimization[J]. SIAM Journal on Numerical Analysis, 2013, 51(2): 927-957.
[16] Pant J K, Lu W S, Antoniou A. New Improved Algorithms for Compressive Sensing Based on Norm[J]. IEEE Transactions on Circuits and Systems II: Express Briefs, 2014, 61(3): 198-202.
[17] Woodworth J, Chartrand R. Compressed sensing recovery via nonconvex shrinkage penalties[J]. Inverse Problems, 2016, 32(7): 075004.
[18] Wu L, Sun Z, Li D H. A BarzilaiโBorwein-Like Iterative Half Thresholding Algorithm for the Regularized Problem[J]. Journal of Scientific Computing, 2016, 67(2): 581-601.
[19] Fengmin X, Shanhe W. A hybrid simulated annealing thresholding algorithm for compressed sensing[J]. Signal Processing, 2013, 93(6): 1577-1585.
[20] Miao C, Yu H. A General-Thresholding Solution for Regularized CT Reconstruction[J]. IEEE Transactions on Image Processing, 2015, 24(12): 5455-5468.
[21] Zhang T. Analysis of multi-stage convex relaxation for sparse regularization[J]. Journal of Machine Learning Research, 2010, 11(Mar): 1081-1107.
[22] Zhang T. Multi-stage convex relaxation for feature selection[J]. Bernoulli, 2013, 19(5B): 2277-2293.
[23] Lou Y, Yin P, Xin J. Point Source Super-resolution Via Non-convex Based Methods[J]. Journal of Scientific Computing, 2016, 68(3): 1082-1100.
[24] Z Zhang S, Xin J. Minimization of transformed penalty: theory, difference of convex function algorithm, and robust application in compressed sensing[J]. Mathematical Programming, 2018, 169(1): 307-336.
[25] Dinh T, Xin J. Convergence of a Relaxed Variable Splitting Method for Learning Sparse Neural Networks via , and transformed- Penalties[J]. arXiv preprint arXiv:1812.05719, 2018.
[26] Lv J, Fan Y. A unified approach to model selection and sparse recovery using regularized least squares[J]. The Annals of Statistics, 2009, 37(6A): 3498-3528.
[27]Bogdan M, Berg E V D, Su W, et al. Statistical estimation and testing via the sorted L1 norm[J]. Statistics, 2013.
[28] Zeng X, Figueiredo M A T. Decreasing Weighted Sorted Regularization[J]. IEEE Signal Processing Letters, 2014, 21(10): 1240-1244.
[29] Lou Y, Yan M. Fast L1โL2 minimization via a proximal operator[J]. Journal of Scientific Computing, 2018, 74(2): 767-785.
[30]Lou, Y., Yin, P., He, Q., Xin, J.: Computing sparse representation in a highly coherent dictionary based on difference of l1 and l2. J. Sci. Comput. 64(1), 178โ196 (2015)
[31] Yin P, Lou Y, He Q, et al. Minimization of L1-2 for compressed sensing[J]. SIAM Journal on Scientific Computing, 2015, 37(1): A536-A563.
[32] Fan J, Li R. Variable selection via nonconcave penalized likelihood and its oracle properties[J]. Journal of the American statistical Association, 2001, 96(456): 1348-1360.
[33] Mehranian A, Rad H S, Rahmim A, et al. Smoothly clipped absolute deviation (SCAD) regularization for compressed sensing MRI using an augmented Lagrangian scheme[J]. Magnetic resonance imaging, 2013, 31(8): 1399-1411.
[34] Zhang C H. Nearly unbiased variable selection under minimax concave penalty[J]. The Annals of statistics, 2010, 38(2): 894-942.
[35] Selesnick I. Sparse regularization via convex analysis[J]. IEEE Transactions on Signal Processing, 2017, 65(17): 4481-4494.
[36] Sun Y, Chen H, Tao J. Sparse signal recovery via minimax-concave penalty and -norm loss function[J]. IET Signal Processing, 2018, 12(9): 1091-1098.
[37] Blumensath T, Davies M E. Iterative thresholding for sparse approximations: The Journal of Fourier Analysis and Applications, 14, 629โ654[J]. 2008.
[38] Blumensath T, Davies M E. Iterative hard thresholding for compressed sensing[J]. Applied and computational harmonic analysis, 2009, 27(3): 265-274.
[39] Blumensath T. Accelerated iterative hard thresholding[J]. Signal Processing, 2012, 92(3): 752-756.
[40] Lu Z. Iterative hard thresholding methods for regularized convex cone programming[J]. Mathematical Programming, 2014, 147(1-2): 125-154.
[41] Bao C, Dong B, Hou L, et al. Image restoration by minimizing zero norm of wavelet frame coefficients[J]. Inverse problems, 2016, 32(11): 115004.
[42] Zhang X, Zhang X. An accelerated proximal iterative hard thresholding method for minimization[J]. arXiv preprint arXiv:1709.01668, 2017..
[43] Gotoh J, Takeda A, Tono K. DC formulations and algorithms for sparse optimization problems[J]. Mathematical Programming, 2018: 1-36.
[44] Tono K, Takeda A, Gotoh J. Efficient DC algorithm for constrained sparse optimization[J]. arXiv preprint arXiv:1701.08498, 2017.
[45] Tao P D, An L T H. Convex analysis approach to dc programming: Theory, algorithms and applications[J]. Acta mathematica vietnamica, 1997, 22(1): 289-355.
[46] Ahn M, Pang J S, Xin J. Difference-of-convex learning: directional stationarity, optimality, and sparsity[J]. SIAM Journal on Optimization, 2017, 27(3): 1637-1665.
[47] Yin P , Xin J . Iterative minimization for non-convex compressed sensing[J]. Journal of Computational Mathematics 2017, 35(4):439-451.
[48] Liu T, Pong T K, Takeda A. A successive difference-of-convex approximation method for a class of nonconvex nonsmooth optimization problems[J]. Mathematical Programming, 2017: 1-29.
[49] Yuille A L, Rangarajan A. The concave-convex procedure[J]. Neural computation, 2003, 15(4): 915-936.
[50] Artacho F J A, Fleming R M T, Vuong P T. Accelerating the DC algorithm for smooth functions[J]. Mathematical Programming, 2018, 169(1): 95-118.
[51] Wen B, Chen X, Pong T K. A proximal difference-of-convex algorithm with extrapolation[J]. Computational optimization and applications, 2018, 69(2): 297-324.
[52] Wen F, Pei L, Yang Y, et al. Efficient and robust recovery of sparse signal and image using generalized nonconvex regularization[J]. IEEE Transactions on Computational Imaging, 2017, 3(4): 566-579.
[53] Zhang J, Zhao C, Zhao D, et al. Image compressive sensing recovery using adaptively learned sparsifying basis via L0 minimization[J]. Signal Processing, 2014, 103: 114-126.
[54] Gong P, Zhang C, Lu Z, et al. A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization problems[C]//International Conference on Machine Learning. 2013: 37-45.
[55] Li H, Lin Z. Accelerated proximal gradient methods for nonconvex programming[C]//Advances in neural information processing systems. 2015: 379-387.
[56] Beck A, Teboulle M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems[J]. SIAM journal on imaging sciences, 2009, 2(1): 183-202.
[57] Dong B, Zhang Y. An efficient algorithm for โ 0 minimization in wavelet frame based image restoration[J]. Journal of Scientific Computing, 2013, 54(2-3): 350-368.
[58] Zhang Y, Dong B, Lu Z. โโ Minimization for wavelet frame based image restoration[J]. Mathematics of Computation, 2013, 82(282): 995-1015.
[59] Zhang X, Lu Y, Chan T. A novel sparsity reconstruction method from Poisson data for 3D bioluminescence tomography[J]. Journal of scientific computing, 2012, 50(3): 519-535.
[60] Trzasko J, Manduca A, Borisch E. Sparse MRI reconstruction via multiscale L0-continuation[C]//2007 IEEE/SP 14th Workshop on Statistical Signal Processing. IEEE, 2007: 176-180.
[61] Trzasko J, Manduca A. Highly Undersampled Magnetic Resonance Image Reconstruction via Homotopic -Minimization[J]. IEEE Transactions on Medical imaging, 2009, 28(1): 106-121.
[62] Pavlikov K, Uryasev S. CVaR norm and applications in optimization[J]. Optimization Letters, 2014, 8(7): 1999-2020.
[63] Gotoh J, Uryasev S. Two pairs of families of polyhedral norms versus -norms: proximity and applications in optimization[J]. Mathematical Programming, 2016, 156(1-2): 391-431.
[64] Sun Y, Chen H, Tao J, et al. Computed tomography image reconstruction from few views via Log-norm total variation minimization[J]. Digital Signal Processing, Volume 88, Pages 172-181, May 2019.
[65] Wen F, Liu P, Liu Y, et al. Robust Sparse Recovery in Impulsive Noise via - Optimization[J]. IEEE Transactions on Signal Processing, 2017, 65(1): 105-118.
[66] Combettes P L, Pesquet J C. Proximal splitting methods in signal processing[M]//Fixed-point algorithms for inverse problems in science and engineering. Springer, New York, NY, 2011: 185-212.
[67] Lu Z. Sequential convex programming methods for a class of structured nonlinear programming[J]. arXiv preprint arXiv:1210.3039, 2012.
[68] Barzilai J, Borwein J M. Two-point step size gradient methods[J]. IMA journal of numerical analysis, 1988, 8(1): 141-148.
[69] Sidky E Y, Chartrand R, Pan X. Image reconstruction from few views by non-convex optimization[C]//2007 IEEE Nuclear Science Symposium Conference Record. IEEE, 2007, 5: 3526-3530.
[70] Rahimi Y, Wang C, Dong H, et al. A Scale Invariant Approach for Sparse Signal Recovery[J]. arXiv preprint arXiv:1812.08852, 2018.
[71]. Nocedal, Jorge, Wright, Stephen J.: Numerical Optimization 2nd. Springer, Berlin (2006)
[72] Boyd S, Parikh N, Chu E, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers[J]. Foundations and Trendsยฎ in Machine learning, 2011, 3(1): 1-122.