Recovery and convergence rate of the Frank-Wolfe Algorithm for the m-EXACT-SPARSE Problem

TL;DR

This paper analyzes the Frank-Wolfe algorithm's ability to efficiently recover sparse signals using a dictionary, proving support-recovery and exponential convergence under certain coherence conditions.

Contribution

It provides theoretical guarantees for support recovery and exponential convergence of the Frank-Wolfe algorithm in sparse reconstruction problems.

Findings

Atoms recruited are limited to the signal's support.

Exponential convergence occurs after a certain iteration.

Support recovery is guaranteed under coherence conditions.

Abstract

We study the properties of the Frank-Wolfe algorithm to solve the m-EXACT-SPARSE reconstruction problem, where a signal y must be expressed as a sparse linear combination of a predefined set of atoms, called dictionary. We prove that when the signal is sparse enough with respect to the coherence of the dictionary, then the iterative process implemented by the Frank-Wolfe algorithm only recruits atoms from the support of the signal, that is the smallest set of atoms from the dictionary that allows for a perfect reconstruction of y. We also prove that under this same condition, there exists an iteration beyond which the algorithm converges exponentially.

Equations

$$\text{find }x\text{ s.t. }y=\Phi x\text{ and }\Vert x{\Vert }_0\le m.$$

$${\mathrm{argmin}}_{x\in {\mathbb{R}}^n}\frac{1}{2}\Vert y-\Phi x{\Vert }_2^2\text{s.t. }\Vert x{\Vert }_0\le m.$$

$$\begin{array}{r}{\mathrm{argmin}}_{x\in {\mathbb{R}}^n}\frac{1}{2}\Vert y-\Phi x{\Vert }_2^2\text{s.t. }\Vert x{\Vert }_1\le \beta \end{array}$$

$${\mu }_1(m)=\underset{|\Lambda |=m}{\max }\underset{i\notin \Lambda }{\max }\sum _{j\in \Lambda }|⟨{\phi }_i,{\phi }_j⟩|,$$

$$\mu ={\mu }_1(1)=\underset{j\ne k}{\max }|⟨{\phi }_j,{\phi }_k⟩|,$$

$$\Vert y-\Phi x_{k+1}{\Vert }_2^2\le \Vert y-\Phi x_k{\Vert }_2^2(1-\theta )\forall k\ge K$$

$$y=\sum _{\stackrel{\lambda \in \Lambda }{|\Lambda |=m}}⟨y,{\phi }_{\lambda }⟩{\phi }_{\lambda },$$

$$\begin{array}{r}\underset{x\in \mathcal{C}}{\min }f(x)\text{s.t. }x\in \mathcal{C}\end{array}$$

$$\begin{array}{r}{\mathrm{argmin}}_{x\in {\mathbb{R}}^n}\frac{1}{2}\Vert y-\Phi x{\Vert }_2^2\text{s.t. }\Vert x{\Vert }_1\le \beta \end{array}$$

$$\underset{x\in \mathcal{C}}{\min }f(x)=\frac{1}{2}\Vert y-\Phi x{\Vert }_2^2,$$

$$\text{s.t. }x\in {\mathcal{B}}_1(\beta )=\{x:\Vert x{\Vert }_1\le \beta \}.$$

$$s_k$$

$$\{\begin{array}{ll}{i}_k& ={\mathrm{argmax}}_{i\in \{1,\dots ,n\}}|⟨{\phi }_i,y-\Phi x_k⟩|\\ s_k& =\mathrm{sign}(⟨{\phi }_{i_k},y-\Phi x_k⟩)\beta e_{i_k}.\end{array}$$

$$r_k=y-\Phi x_k,$$

$$i_k={\lambda }_k={\mathrm{argmax}}_i|⟨{\phi }_i,r_k⟩|.$$

$$f(x_k)=\frac{1}{2}\Vert y-\Phi x_k{\Vert }_2^2=\frac{1}{2}\Vert r_k{\Vert }_2^2.$$

$$\rho (r_k)=\frac{\Vert {\Phi }_{\lambda }^t{r}_k{\Vert }_{\infty }}{\Vert {\Phi }_{opt}^t{r}_k{\Vert }_{\infty }}<1.$$

$$|f(x_k)-f(x^{\ast })|$$

$$=\frac{1}{2}\Vert \Phi x^{\ast }-\Phi x_k{\Vert }_2^2$$

$$\ge \frac{1}{2}{\left({\lambda }_{min}^{\ast }\right)}^2\Vert x^{\ast }-x_k{\Vert }_2^2$$

$$\Vert r_{k+1}{\Vert }_2^2\le \Vert r_k{\Vert }_2^2(1-\theta )$$

$$\theta =\frac{1}{16}\left(\frac{1-{\mu }_1(m-1)}{m}\right){\left(1-\frac{\Vert x^{\ast }{\Vert }_1}{\beta }\right)}^2.$$

$$\beta >2\Vert y{\Vert }_2\sqrt{\frac{m}{1-{\mu }_1(m-1)}},$$

$$\Vert y{\Vert }_2=\Vert \Phi x^{\ast }{\Vert }_2\ge {\lambda }_{min}^{\ast }\Vert x^{\ast }{\Vert }_2\ge \sqrt{1-{\mu }_1(m-1)}\Vert x^{\ast }{\Vert }_2.$$

$$\Vert x^{\ast }{\Vert }_1\le \Vert y{\Vert }_2\sqrt{\frac{m}{1-{\mu }_1(m-1)}}.$$

$$\Vert r_k{\Vert }_2^2\le \Vert r_{k-1}{\Vert }_2^2(1-\theta ).$$

$$\log \Vert r_k{\Vert }_2\le \log \Vert y\Vert +\frac{1}{2}k\log (1-\theta ).$$

$${\gamma }_k=\frac{⟨\Phi (s_k-x_k),r_k⟩}{\Vert \Phi (s_k-x_k){\Vert }_2^2}.$$

$${\gamma }_k={\mathrm{argmin}}_{\gamma \in [0,1]}\Vert y-\Phi (x_k+\gamma (s_k-x_k)){\Vert }_2^2$$

Full text

I Introduction

I-A The m-Exact-Sparse problem

Given a signal $y$ in $\mathbb{R}^{d}$ that is $m$-sparse with respect to some dictionary $\Phi$ in $\mathbb{R}^{d\times n}$, i.e. $y$ is a linear combination of at most $m$ atoms/columns of $\Phi$, the m-Exact-Sparse problem is to find a linear combination of (at most) $m$ atoms that is equal to $y$. The Matching Pursuit (MP) [10] and Orthogonal Matching Pursuit (OMP) [12] algorithms feature nice properties with respect to the m-Exact-Sparse problem, namely recovery and convergence properties. Here, we study these properties for the Frank-Wolfe optimization procedure [6] when used to tackle the m-Exact-Sparse problem.

In the sequel, the dictionary $\Phi=[\varphi_{1}\cdots\varphi_{n}]\in\mathbb{R}^{d\times n}$ is the matrix made of the $n$ atoms $\varphi_{1},\ldots,\varphi_{n}\in\mathbb{R}^{d}$, assumed to be so that $\|\varphi_{i}\|_{2}=1,\forall i$. Here, we consider the case where the support $\Lambda\subseteq\{1,\ldots,n\}$ of the $m$-sparse signal $y$ is unique, the support being the smallest subset of $\{1,\ldots,n\}$ such that $y$ is in the span of the atoms indexed by $\Lambda$ – therefore $|\Lambda|\leq m$. (Section III gives the conditions under which the support is unique.) The sparsity of a linear combination $\Phi x$ ($x\in\mathbb{R}^{n}$) such that $y=\Phi x$ is measured by the number of nonzero entries of $x$, sometimes referred to as the (quasi-)norm $\|{x}\|_{0}$ of $x$.

Formally the m-Exact-Sparse problem is the following. Given a dictionary $\Phi$ and an $m$-sparse signal $y$:

[TABLE]

Since $y$ is a linear combination of at most $m$ atoms of $\Phi$, a solution of Problem (1) can be obtained by solving:

[TABLE]

I-B Related work

The m-Exact-Sparse problem is NP-hard [5], and the ability of a few algorithms that explicitly enforce the $\ell_{0}$-constraint to approximate its solution have been studied, among which brute force methods [11], nonlinear programming approaches [13], and greedy pursuits [10, 12, 14, 15, 1]. Favorable conditions that simultaneously apply to both the sparsity level $m$ and the dictionary $\Phi$ have been exhibited that provide OMP and MP with guarantees on their effectiveness to find exact solutions to m-Exact-Sparse.

Another way to tackle this problem is to recourse to a relaxation strategy by, for instance, replacing the $\ell_{0}$ norm by an $\ell_{1}$ norm. Doing so gives rise to a convex optimization problem, e.g. LASSO [16] and Basis Pursuit [4], that can be handled with a number of provably efficient methods [3]. In addition to the computational benefit of relaxing the problem, Tropp [18] proved that under proper conditions, the supports $\Lambda^{\text{LASSO}}$ and $\Lambda^{\text{BP}}$ of the solutions of the LASSO and Basis Pursuit $\ell_{1}$-relaxations are such that $\Lambda^{\text{LASSO}}\subseteq\Lambda$ and $\Lambda^{\text{BP}}\subseteq\Lambda.$

Here we study the properties of the Frank-Wolfe algorithm [6] to solve the following $\ell_{1}$-relaxation problem:

[TABLE]

where $\|{.}\|_{1}$ is the $\ell_{1}$ norm and $\beta>0$ is a hyperparameter. More precisely, we study how solving (3) may provide an exact solution to (2) and, therefore, to (1).

Going back to the original unrelaxed m-Exact-Sparse problem, we may refine the evoked results regarding the blessed Matching Pursuit (MP) [10] and Orthogonal Matching Pursuit (OMP) [12] greedy algorithms: Tropp [17] and Gribonval and Vandergheynst [7] proved that, if the size $m$ of the support is small enough, then at each iteration, both MP and OMP pick up an atom indexed by the support, thus ensuring recovery properties. Furthermore, as far as convergence is concerned, that is as far as the ability of the procedures to find an exact linear expansion of $y$ is concerned, it was proved that MP shows an exponential rate of convergence, and that OMP reaches convergence after exactly $m$ iterations.

Remark 1**.**

Tropp [17] and Gribonval and Vandergheynst [7] also generalized their results to the case of compressible signals, which are signals that are close to be $m$-sparse. It is beyond the scope of the present paper to consider such signals.

On the other hand, the Frank-Wolfe algorithm [6] is a general-purpose algorithm designed for constrained convex optimization. It has been proven to converge exponentially if the objective function under consideration is strongly convex [8] and linearly in the other cases [6]. As we will see in Section II-C, when solving (3), the atom selection steps in Matching Pursuit and Frank-Wolfe are very similar. This similarity inspired Jaggi and al. [9] to exploit tools used to analyze the Frank-Wolfe algorithm and prove the convergence of the MP algorithm when no assumptions are made on the dictionary.

Here, we go the other way around: we will exploit tools used to analyze MP and OMP to establish properties of the Frank-Wolfe algorithm when seeking a solution of Problem (3).

I-C Main Results

We show that the Frank-Wolfe algorithm, when used to solve (3), enjoys recovery and convergence properties regarding m-Exact-Sparse that are similar to those established in [7, 17] for MP and OMP, under the very same assumptions.

Our results rely on a fundamental quantity associated to a dictionary $\Phi=[\varphi_{1}\cdots\varphi_{n}]$: its Babel function, defined as

[TABLE]

where, by convention: $\mu_{1}(0)=0$. For given $m$, $\mu_{1}(m)$ is roughly a measure on how well any atom from $\Phi$ can be expressed as a linear combination of a set of $m$ other atoms. When $m=1$, the Babel function boils down to

[TABLE]

which is known as the coherence of $\Phi$. In the sequel, we will consider only $m$-sparse signal such that: $m<\tfrac{1}{2}(\mu^{-1}+1)$. In Featured Theorem 1, we state that when $m<\tfrac{1}{2}(\mu^{-1}+1)$, then at each iteration the Frank-Wolfe algorithm picks up an atom indexed by the support of $y$.

Featured Theorem 1**.**

Let $\Phi$ be a dictionary of coherence $\mu$, and $y$ an $m$-sparse signal such that $m<\tfrac{1}{2}(\mu^{-1}+1)$. Then at each iteration, the Frank-Wolfe algorithm picks up an atom of the support of the signal.

Under the same condition, we also prove that the rate of convergence of the Frank-Wolfe algorithm is exponential beyond a certain iteration even though the function we consider is not strongly convex. This is given by Featured Theorem 2.

Featured Theorem 2**.**

Let $\Phi$ be a dictionary of coherence $\mu$, and $y$ an $m$-sparse signal such that $m<\tfrac{1}{2}(\mu^{-1}+1)$. Under some conditions on $y$ and $\beta$, there exists an iteration $K$ of the Frank-Wolfe algorithm and $0<\theta\leq 1$ such that:

[TABLE]

where $\theta$ depends on $\mu_{1}(m-1),\ \beta$, and $y$ and $0<\theta\leq 1$ (which implies the exponential convergence).

I-D Organization of the Paper

In Section II, we instantiate the Frank-Wolfe algorithm for Problem (3) and we relate it to MP and OMP. Section III is devoted to the statement of our main results. We probe their optimality with numerical experiments in Section IV.

II MP, OMP and FW Algorithms

This section recalls Matching Pursuit and Orthogonal Matching Pursuit, the classical greedy algorithms used to tackle m-Exact-Sparse. We then present the Frank-Wolfe algorithm and derive it for Problem (3), showing its similarities with MP and OMP.

II-A Matching Pursuit and Orthogonal Matching Pursuit

Let $\Phi$ be an orthonormal basis and $y$ an exactly $m$-sparse signal (i.e. $y=\Phi x$ with $x$ having exactly $m$ nonzero entries). In this case, the signal can be expressed as

[TABLE]

where $\Lambda$ is the index set of the $m$ atoms $\varphi$ that satisfy: $\langle{y,\varphi}\rangle\neq 0$. The m-Exact-Sparse problem has then an easy solution: one chooses the $m$ atoms having the nonzero inner products with the signal, and the linear expansion of $y$ with respect to these atoms can be obtained readily.

Algorithmically, this can be achieved by building $y_{k}$, the approximation of $y$, one term at a time. Noting $y_{k}$ the current approximation and $r_{k}=y-y_{k}$ the so-called residual, we select at each time step the atom which has the largest inner product (this is a greedy selection) with $r_{k}$, and update the approximation.

MP [10] and OMP [12] are two greedy algorithms used for approximating signals in the general case where the dictionary is not an orthonormal basis. They build upon this idea of greedy selection and iterative updates. MP and OMP initialize the first approximation $y_{0}=0$ and residual $r_{0}=y$ and then repeat the following steps:

Atom selection: $\quad\lambda_{k}=\operatorname{arg\,max}_{i}|{\langle{\varphi_{i},r_{k}}\rangle}|.$ 2. 2.

Approximation update:

1. (a)

MP: $\quad y_{k+1}=y_{k}+\langle{\varphi_{\lambda_{k}},r_{k}}\rangle\varphi_{\lambda_{k}}$ 2. (b)

OMP: $\quad y_{k+1}=\operatorname{arg\,min}_{a\in\operatorname{span}(\{\varphi_{\lambda_{0}},\dots,\varphi_{\lambda_{k}}\})}\|y-a\|_{2}$ 3. 3.

Residual update: $\quad r_{k+1}=y-y_{k+1}.$

II-B Frank-Wolfe

The Frank-Wolfe algorithm [6] is an iterative algorithm developed to solve the optimization problem:

[TABLE]

where $f$ is a convex and continuously differentiable function and $\mathcal{C}$ is a compact and convex set. The Frank-Wolfe algorithm is initialized with an element of $\mathcal{C}$. Then, at iteration $k$, the algorithm applies the three following steps:

Descent direction selection: $\quad s_{k}=\operatorname{arg\,min}_{s\in\mathcal{C}}\langle{s,\nabla f(x_{k})}\rangle.$ 2. 2.

Step size optimization: $\quad\gamma_{k}=\operatorname{arg\,min}_{\gamma\in[0,1]}f((1-\gamma)x_{k}+\gamma s_{k}).$ 3. 3.

Update: $\quad x_{k+1}=(1-\gamma_{k})x_{k}+\gamma_{k}s_{k}.$

Note that the step-size $\gamma_{k}$ can be chosen by other methods [9], without affecting the convergence properties of the algorithm.

II-C Frank-Wolfe for the m-Exact-Sparse problem

We are interested in solving m-Exact-Sparse by finding the solution of the following problem:

[TABLE]

using the Frank-Wolfe algorithm. To this end, we instantiate (4) for Problem (3):

[TABLE]

[TABLE]

$\mathcal{B}_{1}(\beta)$ is the $\ell_{1}$ ball of radius $\beta$; it can be written as the convex hull: $\mathcal{B}_{1}(\beta)=\operatorname{conv}\{\pm\beta e_{i}|i\in\{1,\dots,n\}\}$, with $e_{i}$ the canonical basis vectors of $\mathbb{R}^{n}$. Moreover, $\nabla f(x)=\Phi^{t}(\Phi x-y)$. The selection step of the Frank-Wolfe algorithm thus becomes:

[TABLE]

Since this optimization problem is linear and $\mathcal{B}_{1}(\beta)$ is closed and bounded, there is always an extreme point of $\mathcal{B}_{1}(\beta)$ in the solution set (see [2] for more details), thus:

[TABLE]

or equivalently

[TABLE]

Noticing that $s=\pm\beta e_{i}$ implies $\Phi s=\pm\beta\varphi_{i}$, we conclude that the direction selection step of the Frank-Wolfe algorithm for Problem (3) can be rewritten as:

[TABLE]

Recalling that the residual $r_{k}$ is:

[TABLE]

we notice that we have the same atom selection as in MP and OMP:

[TABLE]

Finally, we specify the initialization $x_{0}=0$ which is in $\mathcal{B}_{1}(\beta)$. This completes the description of the Frank-Wolfe algorithm for Problem (3) which is summarized in Algorithm 1.

In the sequel, we will be interested in the recovery and convergence properties of this algorithm when $m<\tfrac{1}{2}(\mu^{-1}+1)$. This hypothesis implies that the atoms of any subset of at most $m$ atoms ($\{\varphi_{i}|i\in\Lambda\}$ such that $|\Lambda|\leq m$) are necessarily linearly independent and also that for any $m$-sparse signal $y$, the expansion coefficients $x^{*}$ such that $y=\Phi x^{*}$ and $\|{x^{*}}\|_{0}\leq m$ and the corresponding support are unique [17].

In that case, $x^{*}$ is the unique solution of the m-Exact-Sparse problem and also of Problem (2) but not always a solution of Problem (3). Indeed, we always have $f(x^{*})=0$, so if $\|{x^{*}}\|_{1}\leq\beta$ then $x^{*}$ is a solution of Problem (3), but if $\|{x^{*}}\|_{1}>\beta$ then $x^{*}$ is not feasible for Problem (3) hence it is not a solution.

Now, let us clarify some notations. For an $m$-sparse signal $y=\Phi x^{*}$, we denote by $\Lambda_{opt}$ its support i.e. $y=\sum_{i\in\Lambda_{opt}}x^{*}[i]\varphi_{i}$ such that $|\Lambda_{opt}|=m$. For $\Lambda$ a subset of $\{1,\dots,n\}$, we denote by $\Phi_{\Lambda}$ the matrix whose columns are the atoms indexed by $\Lambda$. When $\Lambda$ is the support $\Lambda_{opt}$, we note $\lambda_{min}^{*}$ (resp. $\lambda_{max}^{*}$) its lowest (resp. largest) singular value. For a matrix $\Phi$ we denote by $\operatorname{span}(\Phi)$ the vector space spanned by its columns. Finally, when we study the convergence of Algorithm 1, we consider the residual squared norm $\|{r_{k}}\|^{2}_{2}$, tied to the objective function as follows:

[TABLE]

III Results: Exact recovery and exponential convergence

In this section, we state our main results on the recovery property and the convergence rate of Algorithm 1. We state in Theorem 1 the recovery guarantees of this algorithm, and we present its convergence rate in Theorem 2.

III-A Recovery condition

Tropp [17] proved that when $m<\tfrac{1}{2}(\mu^{-1}+1)$, then OMP exactly recovers the $m$-expansion of any $m$-sparse signal. Gribonval and Vandergheynst [7] proved that the approximated signal constructed by MP algorithm converges to the initial signal. To do so, they prove that at each step, MP and OMP select an atom of the support. Theorem 1 extends this result to the Frank-Wolfe algorithm.

Theorem 1**.**

Let $\Phi$ be a dictionary, $\mu$ its coherence, and $y$ an $m$-sparse signal of support $\Lambda_{opt}$. If $m<\tfrac{1}{2}(\mu^{-1}+1)$, then at each iteration, Algorithm 1 picks up a correct atom, i.e. $\forall\ k$, $i_{k}\in\Lambda_{opt}$.

Remark 2** (ERC).**

As in [17, 7], the condition $m<\tfrac{1}{2}(\mu^{-1}+1)$ can be replaced by a support-specific condition called the exact recovery condition (ERC): $\max\limits_{i\notin\Lambda_{opt}}\|\Phi_{\Lambda_{opt}}^{+}\varphi_{i}\|_{1}<1$, where $\Phi_{\Lambda_{opt}}^{+}$ is the pseudoinverse of the matrix $\Phi_{\Lambda_{opt}}$. ERC is not easy to check in practice because it depends on the unknown support $\Lambda_{opt}$, while the condition $m<\tfrac{1}{2}(\mu^{-1}+1)$, a sufficient condition for ERC [17] to hold, is easy to check.

Proof of Theorem 1.

The proof of this theorem is very similar to the proof of Theorem 3.1 in [17]. One shows by induction that at each step the residual $r_{k}=y-\Phi x_{k}$ remains in $\operatorname{span}(\Phi_{\lambda_{opt}})$ and in the process that the selected atom is in $\Lambda_{opt}$.

• •

$k=0$: by definition $r_{0}=y$ is in $\operatorname{span}(\Phi_{\lambda_{opt}})$.

• •

If $k\geq 0$: we assume that $r_{k}\in\operatorname{span}(\Phi_{\Lambda_{opt}})$. Let $\lambda=\{1,\dots,n\}\setminus\lambda_{opt}$ be the set of atoms which are not in the support of the signal. The atom $\varphi_{i_{k}}$ is a “good atom” (i.e. $i_{k}\in\Lambda_{opt}$), if and only if:

[TABLE]

Tropp [17] proved that if $r_{k}\in\operatorname{span}(\Phi_{\Lambda_{opt}})$ then $\rho(r_{k})\leq\max\limits_{i\notin\Lambda_{opt}}\|\Phi_{\Lambda_{opt}}^{+}\varphi_{i}\|_{1}\leq\frac{\mu_{1}(m)}{1-\mu_{1}(m-1)}$, where $\Phi_{\Lambda_{opt}}^{+}$ is the pseudoinverse of the matrix $\Phi_{\Lambda_{opt}}$. He also proved that $m<\tfrac{1}{2}(\mu^{-1}+1)$ implies $\frac{\mu_{1}(m)}{1-\mu_{1}(m-1)}<1$.

Hence $i_{k}$ is in $\Lambda_{opt}$ and $s_{k}=\pm\beta e_{i_{k}}$ is thus in $\operatorname{span}(\Phi_{\Lambda_{opt}})$. Since $r_{k+1}=r_{k}-\gamma_{k}\Phi(s_{k}-x_{k})$, and since by assumption $r_{k}$ is also in $\operatorname{span}(\Phi_{\Lambda_{opt}})$, we deduce that $r_{k+1}$ is in $\operatorname{span}(\Phi_{\Lambda_{opt}})$.

∎

Theorem 1 specifies that if the signal has a sparsity $m$ smaller than $\tfrac{1}{2}(\mu^{-1}+1)$, Algorithm 1 only recruits atoms of the support. As noted is Section II-C, the expansion $x^{*}$ can not always be reached (because it might be the case that $\|{x^{*}}\|_{1}>\beta$). In the case when it can be reached (i.e. when $\|{x^{*}}\|_{1}\leq\beta$) one can furthermore prove that the expansion $x^{*}$ itself is recovered:

Corollary 1**.**

Let $\Phi$ be a dictionary, $\mu$ its coherence, and $y$ an $m$-sparse signal of support $\Lambda_{opt}$. If $m<\tfrac{1}{2}(\mu^{-1}+1)$ and $\|{x^{*}}\|_{1}\leq\beta$ then the sequence $x_{k}$ converges to $x^{*}$ (i.e. Algorithm 1 exactly recovers the $m$-expansion of $y$).

Proof of Corollary 1.

$\|{x^{*}}\|_{1}\leq\beta$ so that $x^{*}$ is a solution of Problem (3). The Frank-Wolfe algorithm is known to converge in terms of objective values ($f(x_{k})$), we deduce that $f(x_{k})$ converges to $f(x^{*})=0$. Since Theorem 1 ensures that the iterates $x_{k}$ are in $\operatorname{span}(\Phi_{\Lambda_{opt}})$, we also have convergence of the iterates ($x_{k}$ converge to $x^{*}$) since

[TABLE]

where the last line holds because $x_{k}-x^{*}$ is in $\operatorname{span}(\Phi_{\Lambda_{opt}})$ and $\lambda_{min}^{*}>0$ since $m<\tfrac{1}{2}(\mu^{-1}+1)$ [17]. ∎

In this section, we presented the recovery guarantees for the Frank-Wolfe algorithm. In the next section, we will show that the convergence rate of the Frank-Wolfe algorithm is exponential when $m<\tfrac{1}{2}(\mu^{-1}+1)$ and $\beta$ is large enough so that the expansion $x^{*}$ is recovered.

III-B Rate of convergence

As mentioned in the introduction, in the generic case of Problem (4), the Frank-Wolfe algorithm converges exponentially beyond a certain iteration when the objective function is strongly convex [8] and linearly in the other cases [6]. We prove in Theorem 2, that when $m<\tfrac{1}{2}(\mu^{-1}+1)$, the Frank-Wolfe algorithm converges exponentially beyond a certain iteration even though the function we consider is not strongly convex.

Theorem 2**.**

*Let $\Phi$ be a dictionary, $\mu$ its coherence, $\mu_{1}$ its Babel function, and $y=\Phi x^{*}$ an $m$-sparse signal.

If $m<\tfrac{1}{2}(\mu^{-1}+1)$ and $\|{x^{*}}\|_{1}<\beta$, then there exists $K$ such that for all iterations $k\geq K$ of Algorithm 1, we have:*

[TABLE]

where

[TABLE]

The proof of Theorem 2 is available in Appendix A.

Remark 3**.**

Note that $0<\theta\leq 1$. Indeed, $m<\tfrac{1}{2}(\mu^{-1}+1)$ implies $0<\mu_{1}(m-1)<1$ so that $0<\tfrac{1}{16m}(1-\mu_{1}(m-1))\left(1-\tfrac{\|{x^{*}}\|_{1}}{\beta}\right)^{2}\leq 1$ i.e. $0<\theta\leq 1$. Thus, Theorem 2 shows exponential convergence.

Remark 4**.**

As for Theorem 1, the same result holds when the Exact Recovery Condition (ERC), $\mu_{1}(m-1)<1$ and $\|{x^{*}}\|_{1}<\beta$ hold.

Remark 5**.**

As we said above, the objective function that we consider is not strongly convex, but since the constructed iterates $x_{k}$ remains in the $\operatorname{span}(\Phi_{\Lambda_{opt}})$, the function takes the form of a $\lambda_{min}^{*}$-strongly convex function.

A natural question that comes from Theorem 2 and from the convergence rate of MP and OMP is whether it is possible to guarantee the exponential convergence from the first iteration. The following theorem proves that this is possible if $\beta$ is large enough.

Theorem 3**.**

Let $\Phi$ be a dictionary of coherence $\mu$ and $y$ an $m$-sparse signal. If $m<\tfrac{1}{2}(\mu^{-1}+1)$ and

[TABLE]

then Algorithm 1 converges exponentially from the first iteration.

The proof of Theorem 3 is also available in Appendix B.

In the proof of Theorem 2 (see Appendix A), we show that when the iterates $x_{k}$ stay close enough to $x^{*}$ ($\|{x_{k}-x^{*}}\|_{1}\leq\epsilon=\frac{\beta-\|{x^{*}}\|_{1}}{2}$), Algorithm 1 converges exponentially. The intuition of Theorem 3 is to choose $\beta$ large enough so that a similar bound is guaranteed from the first iteration.

Remark 6**.**

Let us remark that the assumption $\beta>2\|{y}\|_{2}\sqrt{\tfrac{m}{1-\mu_{1}(m-1)}}$ is stronger than that of Theorem 2 ($\beta>\|{x^{*}}\|_{1}$). Indeed, we have on the one hand: $\|{x^{*}}\|_{1}\leq\sqrt{m}\|{x^{*}}\|_{2}$ because $x^{*}$ has non-zero coefficients only in $\Lambda_{opt}$. On the other hand:

[TABLE]

We conclude

[TABLE]

So that $\beta>2\|{y}\|_{2}\sqrt{\tfrac{m}{1-\mu_{1}(m-1)}}$ implies $\beta>\|{x^{*}}\|_{1}$.

Besides, while the assumption of Theorem 2 ($\beta>\|{x^{*}}\|_{1}$) can not be verified beforehand since it depends on the unknown $x^{*}$, the assumption of Theorem 3 can be checked since it depends on the dictionary and $y$.

IV Numerical Simulations

Theorem 2 shows that there is exponential convergence when the sparsity $m$ is small enough ($m<\tfrac{1}{2}(\mu^{-1}+1)$) and $\beta$ is larger than $\|{x^{*}}\|_{1}$. The goal of this section is to investigate whether these conditions are tight by performing three numerical experiments on synthetic data.

We simulate in Python signals of size $d=10000$ that are sparse on a dictionary of $n=20000$ atoms. The considered dictionaries are random, with coefficients following a standard normal distribution. The mean coherence of such dictionaries is around $\mu=5.8\times 10^{-2}$, and $m^{*}=\lceil\tfrac{1}{2}(\mu^{-1}-1)\rceil=8$ is the largest integer such that the condition of Theorem 2 holds. The signals are also random. The supports of size $m$ are drawn uniformly at random while the corresponding coefficients are chosen using a standard normal distribution. For each experiment, Algorithm 1 is run for $2000$ simulated signals and dictionaries.

The exponential convergence in Theorem 2, is quantified by

[TABLE]

which is equivalent to

[TABLE]

In the first two experiments, we visualize the convergence rate by displaying the quantity $\log\|{r_{k}}\|_{2}$, the convergence being exponential when this is upper-bounded by a line with negative slope (the steepest the slope, the fastest the convergence).

In the first experiment, the values of $m$ and $\beta$ comply with the conditions of Theorem 2. We fix $\beta=8\|{x^{*}}\|_{1}$, and $m=m^{*}$. We draw in Figure 1 the mean and the maximum over the $2000$ simulated signals and dictionaries of the function $\log\|{r_{k}}\|_{2}$ for each iteration $k$, and compare it to the theoretical bound in Theorem 2. As expected, the maximum and the mean of the function $\log\|{r_{k}}\|_{2}$ can be bounded above by a line with negative slope, and thus converge exponentially. We also notice that in practice, the maximum and the mean are much lower than the theoretical prediction. This suggests that the theoretical bound might be improved in this case.

In the second experiment, we investigate if the exponential convergence is still possible when the sparsity is larger than $m^{*}=\lceil\tfrac{1}{2}(\mu^{-1}+1)\rceil-1$, i.e., when the condition of Theorem 2 is not satisfied. We fix here $\beta=8\|{x^{*}}\|_{1}$ and show in Figure 2 the maximal value of $\log\|{r_{k}}\|_{2}$ for $m=m^{*}$, $2m^{*}$, $5m^{*}$ and $20m^{*}$ on $2000$ signals and dictionaries. We observe that exponential convergence still arises at least up to $m=5m^{*}$ but probably not for $m=20m^{*}$, suggesting that in practice one may reconstruct very fast a larger set of signals than only those being $m^{*}$-sparse, and that there might be room for a little improvement in the assumption $m\leq m^{*}$ in Theorem 2.

In the last experiment, we study the influence of the distance from $x^{*}$ to $\mathcal{B}_{1}(\beta)$ on the convergence rate. Indeed Theorem 2 predicts that the convergence slows down when $\|{x^{*}}\|_{1}$ approaches $\beta$ and does not predict exponential convergence if $\|{x^{*}}\|_{1}=\beta$. In this experiment, the sparsity $m$ is fixed to $m=m^{*}$. We show in Figure 3 the mean and theoretical values of $\log\|{r_{k}}\|_{2}$ on $2000$ signals and dictionaries in two cases: either $\beta=\beta_{1}=1.1\|{x^{*}}\|_{1}$ or $\beta=\beta_{2}=8\|{x^{*}}\|_{1}$. As expected, the negative slope is steeper when $\beta$ is larger. For large $\beta$ ($\beta_{2}$), the slope stays well below the slope predicted by the theoretical bound. This is not the case anymore for $\beta$ close to $\|{x^{*}}\|_{1}$ ($\beta_{1}$), where the curve becomes horizontal, suggesting that the theoretical bound may be reached and that the assumption $\beta>\|{x^{*}}\|_{1}$ might be necessary.

V Conclusion

We study the properties of the Frank-Wolfe algorithm when solving the m-Exact-Sparse problem and we prove that, like with MP and OMP, when the signal is sparse enough with respect to the coherence of the dictionary, the Frank-Wolfe algorithm picks up only atoms of the support. We also prove that under this same condition, the Frank-Wolfe algorithm converges exponentially. In the experimental part, we have observed the optimality of the obtained bound in terms of the size of the $\ell_{1}$-ball constraining the search space, gaining some insights on the sparsity bound, that would suggest studying its tightness in future work. Extending these results to the case of non-exact-sparse but only compressible signals is also a natural next step.

Appendix A Proof of Theorem 2

To prove Theorem 2, we need to introduce the following lemma:

Lemma 1**.**

For any iteration $k$ of Algorithm 1, if $\gamma_{k}\neq 0$, and if $\|{y}\|_{2}<\beta$ then:

[TABLE]

Proof.

Recall that

[TABLE]

and define

[TABLE]

Note that

[TABLE]

so we wish to prove that $\gamma_{k}=\gamma^{*}_{k}$.

Because $\gamma_{k}$ is the solution of the same minimization problem as $\gamma^{*}_{k}$ but restricted on the interval $[0,1]$, we have only three possibilities: (i) $\gamma^{*}_{k}\geq 1$ and $\gamma_{k}=1$, (ii) $0<\gamma_{k}=\gamma^{*}_{k}<1$, (iii) $\gamma^{*}_{k}\leq 0$ and $\gamma_{k}=0$. Here we assume that $\gamma_{k}\not=0$ so the last possibility (iii) is ruled out. What is left to do to finish the proof is to rule out the first possibility: (i) $\gamma^{*}_{k}\geq 1$ and $\gamma_{k}=1$.

To do so, consider these two different cases:

• •

$k=0$: since $x_{0}=0$ and $r_{0}=y$,

[TABLE]

Since $\|{y}\|_{2}<\beta$, we have $\gamma_{0}^{*}<1$. Moreover, by construction of $s_{0}$, $\langle{\Phi s_{0},y}\rangle>0$ so $0<\gamma_{0}^{*}<1$. We conclude we are in case (ii) and $\gamma_{0}=\gamma_{0}^{*}$.

• •

$k\neq 0$: assume that $\gamma_{k}=1$. We then have $x_{k+1}=s_{k}$. By construction of the Frank-Wolfe algorithm we have:

[TABLE]

Since we proved that $\gamma_{0}\neq 1$ , we have:

[TABLE]

This implies $f(s_{k})<f(s_{0})$, that is:

[TABLE]

Since $s_{0}=\operatorname{sign}(\langle{\varphi_{i_{0}},y}\rangle)\beta e_{i_{0}}$, both sides of the previous equation are positive:

[TABLE]

This is clearly a contradiction because $s_{0}=\operatorname{arg\,max}_{s\in\mathcal{B}_{1}(\beta)}\langle{\Phi s,y}\rangle$. We conclude that $0<\gamma_{k}<1$ so that we are again in case (ii) where $\gamma_{k}=\gamma_{k}^{*}$.

We conclude that if $\gamma_{k}>0$ and $\|{y}\|_{2}<\beta$ then $\gamma_{k}=\frac{\langle{\Phi(s_{k}-x_{k}),r_{k}}\rangle}{\|{\Phi(s_{k}-x_{k})}\|^{2}_{2}}$. ∎

Proof of Theorem 2.

Note that we are in the case where both Theorem 1 and Corollary 1 hold. Let $k$ be an iteration of Algorithm 1. There are two possibles values for $\gamma_{k}$:

$\gamma_{k}=0$

then $x_{k}=x_{k+1}$ and subsequently for all $l\geq k$, $x_{k}=x_{l}$ and $f(x_{k})=f(x_{l})$. The convergence of the objective values yields: $f(x_{l})=f(x^{*})=0$ for $l\geq k$ and in particular $\|{r_{k+1}}\|^{2}_{2}=\|{r_{k}}\|^{2}_{2}=0$. Thus Theorem 2 holds.

$0<\gamma_{k}\leq 1$

by definition of the residual, we have:

[TABLE]

$\gamma_{k}$ is the solution of the following optimization problem:

[TABLE]

Notice that $\|{\Phi v}\|_{2}\leq\|{v}\|_{1}$ for all vectors $v$ in $\mathbb{R}^{n}$ since the $\varphi_{i}$ are of unit norm ($\|{\Phi v}\|_{2}^{2}=\sum_{i,j=1}^{n}v[i]v[j]\langle{\varphi_{i},\varphi_{j}}\rangle\leq\sum_{i,j=1}^{n}|v[i]||v[j]|=\|{v}\|_{1}^{2}$). Hence

[TABLE]

As showed in Lemma 1, the solution of the previous problem is then:

[TABLE]

Replacing the value of $\gamma_{k}$ in the previous equation, we obtain:

[TABLE]

We shall now lower-bound $\langle{\Phi(s_{k}-x_{k}),r_{k}}\rangle^{2}$ and upper-bound $\|{\Phi(s_{k}-x_{k})}\|^{2}_{2}$.

To bound $\|{\Phi(s_{k}-x_{k})}\|_{2}^{2}$, we use $\|{\Phi v}\|_{2}\leq\|{v}\|_{1}$ and $s_{k}$ and $x_{k}$ are in $\mathcal{B}_{1}(\beta)$:

[TABLE]

To bound $\langle{\Phi(s_{k}-x_{k}),r_{k}}\rangle^{2}$, fix $\epsilon=\tfrac{\beta-\|{x^{*}}\|_{1}}{2}>0$. As noted in Corollary 1, the iterates $x_{k}$ converge to $x^{*}$, there exists an iteration $K$ such that for every $k\geq K$: $\|{x_{k}-x^{*}}\|_{1}\leq\epsilon$. Fix $k\geq K$. Let us define $u\in\mathbb{R}^{n}$, such that:

[TABLE]

One can show that $x_{k}+u$ is in $\mathcal{B}_{1}(\beta)$, indeed

[TABLE]

Noting that $\|{\Phi_{\Lambda_{opt}}^{t}r_{k}}\|_{1}\leq\sqrt{m}\|{\Phi_{\Lambda_{opt}}^{t}r_{k}}\|_{2}$ (because $\Phi_{\Lambda_{opt}}^{t}r_{k}\in\mathbb{R}^{|\Lambda_{opt}|}$ and $|\Lambda_{opt}|\leq m$) leads to:

[TABLE]

We conclude that $x_{k}+u$ is in $\mathcal{B}_{1}(\beta)$.

Since $s_{k}=\operatorname{arg\,min}_{s\in\mathcal{B}_{1}(\beta)}\langle{s,\nabla f(x_{k})}\rangle$ then:

[TABLE]

By definition of $f$: $\nabla f(x_{k})=-\Phi^{t}r_{k}$, thus:

[TABLE]

Noting that

[TABLE]

we conclude:

[TABLE]

By Theorem 1, $r_{k}$ is in $\operatorname{span}(\Phi_{\Lambda_{opt}})$ and since the atoms indexed by $\Lambda_{opt}$ are linearly independent (thus $\lambda_{min}^{*}>0$), we obtain:

[TABLE]

and

[TABLE]

By Lemma $2.3$ of [17], $(\lambda_{min}^{*})^{2}\geq 1-\mu_{1}(m-1)$. Since $m<\tfrac{1}{2}(\mu^{-1}+1)$ implies $\mu_{1}(m)+\mu_{1}(m-1)<1$ [17], we have $0<1-\mu_{1}(m-1)<1$ and deduce that:

[TABLE]

Plugging in the bounds of Eq. (7) and (8) in Eq. (6), we obtain:

[TABLE]

which finishes the proof. ∎

Appendix B Proof of Theorem 3

To prove Theorem 3, the key is to bound uniformly the $l_{1}$ norm of the iterates $x_{k}$. This is the purpose of the following lemma.

Lemma 2**.**

Let $\Phi$ be a dictionary, $\mu$ its coherence, and $y=\Phi x^{*}$ an $m$-sparse signal. If $m<\tfrac{1}{2}(\mu^{-1}+1)$, then for each iteration $k$ of Algorithm 1

[TABLE]

Proof.

Indeed, we have on the one hand: $\|{x_{k}}\|_{1}\leq\sqrt{m}\|{x_{k}}\|_{2}$ because $x_{k}$ has non-zero coefficients only in $\lambda_{opt}$ (proved in Theorem 1). On the other hand:

[TABLE]

We conclude

[TABLE]

Moreover

[TABLE]

where the third line holds because by construction of the Frank-Wolfe algorithm, the sequence $\{f(x_{k})\}_{k}$ is non increasing. So we conclude that $\|{x_{k}}\|_{1}\leq 2\|{y}\|_{2}\sqrt{\tfrac{m}{1-\mu_{1}(m-1)}}$ for all $k$. ∎

Proof of Theorem 3.

Since $\beta>2\|{y}\|_{2}\sqrt{\tfrac{m}{1-\mu_{1}(m-1)}}\geq\|{x^{*}}\|_{1}$ (see Remark 6), we can re-use the arguments of the proof of Theorem 2. We will do so up to Eq. (7) and only modify the lower bound on $\langle{\Phi(s_{k}-x_{k}),r_{k}}\rangle^{2}$.

By definition of $s_{k}$,

[TABLE]

and

[TABLE]

We conclude that:

[TABLE]

By Lemma 2 of [7]:

[TABLE]

Hence:

[TABLE]

What is left to prove is that $1-\frac{\|{x_{k}}\|_{1}}{\beta}$ is uniformly bounded away from zero. Using Lemma 2 we obtain:

[TABLE]

By assumption $1-\frac{2\|{y}\|_{2}}{\beta}\sqrt{\tfrac{m}{1-\mu_{1}(m-1)}}>0$, we deduce that for all $k$:

[TABLE]

with

[TABLE]

Using this and Eq. (6), we obtain:

[TABLE]

We conclude that for all $k$:

[TABLE]

with $0<\left(1-\frac{(1-\mu_{1}(m-1))}{4m}\left(1-\tau\right)^{2}\right)<1$ which proves the exponential convergence from the first iteration. ∎

Acknowledgment

The authors would like to thank the Agence Nationale de la Recherche under grant JCJC MAD (ANR-14-CE27-0002) which supported this work.