Description of the minimizers of least squares regularized~ with~ $\bm{\ell_0}$-norm. Uniqueness of the global minimizer

TL;DR

This paper analyzes the minimizers of a least squares problem with an $m{ ext{ extlbrackdbl}0 ext{ extrbrackdbl}}$-norm penalty, providing theoretical insights into their uniqueness, structure, and sparsity properties, with implications for algorithm development.

Contribution

It offers a comprehensive theoretical analysis of local and global minimizers of the $m{ ext{ extlbrackdbl}0 ext{ extrbrackdbl}}$-penalized least squares problem, including conditions for uniqueness and sparsity.

Findings

Strict local minimizers are easy to compute without knowing $eta$.

Global minimizers are always strict and can be characterized under rank assumptions.

For large $eta$, all global minimizers are $ ext{ extlbrackdbl}0, ext{ extrbrackdbl}$-sparse, with uniqueness outside negligible data sets.

Abstract

We have an $\m\x\n$ real-valued arbitrary matrix $A$ (e.g. a dictionary) with $\m<\n$ and data $d$ describing the sought-after object with the help of $A$. This work provides an in-depth analysis of the (local and global) minimizers of an objective function $\Fd$ combining a quadratic data-fidelity term and an $\ell_0$ penalty applied to each entry of the sought-after solution, weighted by a regularization parameter $\be>0$. For several decades, this objective has attracted a ceaseless effort to conceive algorithms approaching a good minimizer. Our theoretical contributions, summarized below, shed new light on the existing algorithms and can help the conception of innovative numerical schemes. To solve the normal equation associated with any $\m$-row submatrix of $A$ is equivalent to compute a local minimizer $\hu$ of $\Fd$. (Local) minimizers $\hu$ of $\Fd$ are strict if and only if the submatrix, composed of those columns of $A$ whose indexes form the support of $\hu$, has full column rank. An outcome is that strict local minimizers of $\Fd$ are easily computed without knowing the value of $\be$. Each strict local minimizer is linear in data. It is proved that $\Fd$ has global minimizers and that they are always strict. They are studied in more details under the (standard) assumption that $\rank(A)=\m<\n$. The global minimizers with $\m$-length support are seen to be impractical. Given $d$, critical values $\be_\k$ for any $\k\leq\m-1$ are exhibited such that if $\be>\be_\k$, all global minimizers of $\Fd$ are $\k$-sparse. An assumption on $A$ is adopted and proved to fail only on a closed negligible subset. Then for all data $d$ beyond a closed negligible subset, the objective $\Fd$ for $\be>\be_\k$, $\k\leq\m-1$, has a unique global minimizer and this minimizer is $\k$-sparse. Instructive small-size ($5\x 10$) numerical illustrations confirm the main theoretical results.