On optimal solutions of the constrained $\ell_0$ regularization and its penalty problem

TL;DR

This paper systematically studies the solutions of constrained  regularization and its penalty problem, revealing their relationships, stability, and conditions for equivalence, with implications for sparse reconstruction.

Contribution

It provides a comprehensive analysis of the optimal solutions and their connections, including conditions for equivalence and stability, which were not previously well-understood.

Findings

Optimal solution set of the penalty problem is piecewise constant w.r.t. penalty parameter.

In noisy cases, the penalty problem often has no common solutions with the constrained  problem.

Sufficiently large penalty parameters can ensure the same solutions for both problems.

Abstract

The constrained $\ell_0$ regularization plays an important role in sparse reconstruction. A widely used approach for solving this problem is the penalty method, of which the least square penalty problem is a special case. However, the connections between global minimizers of the constrained $\ell_0$ problem and its penalty problem have never been studied in a systematic way. This work provides a comprehensive investigation on optimal solutions of these two problems and their connections. We give detailed descriptions of optimal solutions of the two problems, including existence, stability with respect to the parameter, cardinality and strictness. In particular, we find that the optimal solution set of the penalty problem is piecewise constant with respect to the penalty parameter. Then we analyze in-depth the relationship between optimal solutions of the two problems. It is shown that, in the noisy case the least square penalty problem probably has no common optimal solutions with the constrained $\ell_0$ problem for any penalty parameter. Under a mild condition on the penalty function, we establish that the penalty problem has the same optimal solution set as the constrained $\ell_0$ problem when the penalty parameter is sufficiently large. Based on the conditions, we further propose exact penalty problems for the constrained $\ell_0$ problem. Finally, we present a numerical example to illustrate our main theoretical results.