On the stable recovery of deep structured linear networks under sparsity constraints

TL;DR

This paper establishes sharp conditions for the stable recovery of deep structured linear networks with sparsity constraints, ensuring that parameter errors scale linearly with reconstruction errors, thus enabling reliable feature interpretation and statistical guarantees.

Contribution

It introduces a novel stability analysis for deep linear networks with sparsity priors using Tensorial Lifting, providing sharp guarantees especially for convolutional networks.

Findings

Stability error scales linearly with reconstruction error under certain conditions.

Sparsity prior improves the stability constant.

Analysis distinguishes effects of architecture and input data on stability.

Abstract

We consider a deep structured linear network under sparsity constraints. We study sharp conditions guaranteeing the stability of the optimal parameters defining the network. More precisely, we provide sharp conditions on the network architecture and the sample under which the error on the parameters defining the network scales linearly with the reconstruction error (i.e. the risk). Therefore, under these conditions, the weights obtained with a successful algorithms are well defined and only depend on the architecture of the network and the sample. The features in the latent spaces are stably defined. The stability property is required in order to interpret the features defined in the latent spaces. It can also lead to a guarantee on the statistical risk. This is what motivates this study. The analysis is based on the recently proposed Tensorial Lifting. The particularity of this paper is to consider a sparsity prior. This leads to a better stability constant. As an illustration, we detail the analysis and provide sharp stability guarantees for convolutional linear network under sparsity prior. In this analysis, we distinguish the role of the network architecture and the sample input. This highlights the requirements on the data in connection to parameter stability.

Equations

$$\sum _{i=1}^n\Vert f_{\mathbf{w}}(x_i)-y_i{\Vert }^2\le \eta \text{mbox}and\sum _{i=1}^n\Vert f_{{\mathbf{w}}^{\prime }}(x_i)-y_i{\Vert }^2\le \eta$$

$$d(\mathbf{w},{\mathbf{w}}^{\prime })\le C\eta .$$

$$\sum _{i=1}^n\Vert f_{\mathbf{w}}(x_i)-y_i{\Vert }^2$$

$$\sum _{i=1}^n\Vert f_{{\mathbf{w}}^{\prime }}(x_i)-y_i{\Vert }^2$$

$$M_h:{\mathbb{R}}^S$$

$$w$$

$$f_{\mathbf{w}}(x)=M_H({\mathbf{w}}_H)M_{H-1}({\mathbf{w}}_{H-1})\cdots M_2({\mathbf{w}}_2)M_1({\mathbf{w}}_1)x.$$

$$\mathrm{supp}\left(\mathbf{w}\right)\subset \mathcal{S}$$

$$\mathrm{supp}\left(\overline{\mathbf{w}}\right)\subset \overline{\mathcal{S}}\text{mbox}and\mathrm{supp}\left({\overline{\mathbf{w}}}^{\prime }\right)\subset {\overline{\mathcal{S}}}^{\prime }$$

$$\Vert f_{\overline{\mathbf{w}}}(X)-Y\Vert =\delta \text{mbox}and\Vert f_{{\overline{\mathbf{w}}}^{\prime }}(X)-Y\Vert =\eta$$

$$\prod _{h=1}^H{\lambda }_h=1\text{mbox}and\forall h=\mathrm{1..}H,{\mathbf{w}}_h={\lambda }_h{\mathbf{v}}_h.$$

$$d_p([\mathbf{w}],[\mathbf{v}])=\underset{\begin{array}{c}{\mathbf{w}}^{\prime }\in [\mathbf{w}]\cap {\mathbb{R}}_{\text{mbox}diag}^{S\times H}\\ {\mathbf{v}}^{\prime }\in [\mathbf{v}]\cap {\mathbb{R}}_{\text{mbox}diag}^{S\times H}\end{array}}{\inf }\Vert {\mathbf{w}}^{\prime }-{\mathbf{v}}^{\prime }{\Vert }_p,\forall \mathbf{w},\mathbf{v}\in {\mathbb{R}}_{\ast }^{S\times H},$$

$${\mathbb{R}}_{\text{mbox}diag}^{S\times H}=\{\mathbf{w}\in {\mathbb{R}}_{\ast }^{S\times H}|\forall h=\mathrm{1..}H,\Vert {\mathbf{w}}_h{\Vert }_{\infty }=\Vert {\mathbf{w}}_1{\Vert }_{\infty }\}.$$

$$T_{\mathbf{i}}={\mathbf{w}}_{1,i_1}\dots {\mathbf{w}}_{H,i_H}.$$

$$\begin{array}[]{rcl}P:\mathbb{R}^{S\times H}&\longrightarrow&\Sigma_{1}\subset\mathbb{R}^{S^{H}}\\ \mathbf{w}&\longmapsto&(\mathbf{w}_{1,i_{1}}\mathbf{w}_{2,i_{2}}\ldots\mathbf{w}_{H,i_{H}})_{\mathbf{i}\in[\![S]\!]^{H}}\end{array}$$

$$d_p([\mathbf{w}],[{\mathbf{w}}^{\prime }])\le 7(HS{)}^{\frac{1}{p}}\min \left(\Vert P(\mathbf{w}){\Vert }_{\infty }^{\frac{1}{H}-1},\Vert P({\mathbf{w}}^{\prime }){\Vert }_{\infty }^{\frac{1}{H}-1}\right)\Vert P(\mathbf{w})-P({\mathbf{w}}^{\prime }){\Vert }_q.$$

$$\Vert P(\mathbf{w})-P({\mathbf{w}}^{\prime }){\Vert }_q\le S^{\frac{H-1}{q}}{H}^{1-\frac{1}{q}}\max \left(\Vert P(\mathbf{w}){\Vert }_{\infty }^{1-\frac{1}{H}},\Vert P({\mathbf{w}}^{\prime }){\Vert }_{\infty }^{1-\frac{1}{H}}\right)d_q([\mathbf{w}],[{\mathbf{w}}^{\prime }]).$$

$$\mathcal{A}:{\mathbb{R}}^{S^H}⟶{\mathbb{R}}^{m\times n},$$

$$M_H({\mathbf{w}}_H)\cdots M_1({\mathbf{w}}_1)X=\mathcal{A}P(\mathbf{w}).$$

$${\mathbb{R}}_{\mathcal{S}}^{S\times H}=\{\mathbf{w}\in {\mathbb{R}}^{S\times H}|{\mathbf{w}}_{h,i}=0\text{mbox},forallh=\mathrm{1..}H\text{mbox}andi\notin {\mathcal{S}}_h\}$$

$${\mathbb{R}}_{\mathcal{S}}^{S^H}=\{T\in {\mathbb{R}}^{S^H}|T_{\mathbf{i}}=0\text{mbox},if\exists h=\mathrm{1..}H\text{mbox},suchthat{\mathbf{i}}_h\notin {\mathcal{S}}_h\}.$$

$$({\mathbf{P}}_{\mathcal{S}}T)_{\mathbf{i}}=\left\{\begin{array}[]{ll}{T}_{\mathbf{i}}&\mbox{, if }\mathbf{i}\in\mathcal{S},\\ 0&\mbox{, otherwise.}\end{array}\right.$$

$${\mathcal{A}}_{\mathcal{S}}=\mathcal{A}{\mathbf{P}}_{\mathcal{S}}.$$

$${\mathcal{A}}_{\mathcal{S}\cup {\mathcal{S}}^{\prime }}P(\mathbf{w})$$

$$\Vert T\Vert \le \gamma \Vert T-{\mathbf{P}}_{\mathcal{S}\cup {\mathcal{S}}^{\prime }}{T}^{\prime }\Vert .$$

$${\sigma }_{\mathcal{M}}\Vert {\mathbf{P}}_{\mathcal{S}\cup {\mathcal{S}}^{\prime }}T\Vert \le \Vert {\mathcal{A}}_{\mathcal{S}\cup {\mathcal{S}}^{\prime }}T\Vert .$$

$$\Vert {\mathcal{A}}_{\mathcal{S}\cup {\mathcal{S}}^{\prime }}T\Vert \ge {\sigma }_{\mathcal{S}\cup {\mathcal{S}}^{\prime }}\Vert T\Vert \ge {\sigma }_{\mathcal{S}\cup {\mathcal{S}}^{\prime }}\Vert {\mathbf{P}}_{\mathcal{S}\cup {\mathcal{S}}^{\prime }}T\Vert ,$$

$$\Vert P({\overline{\mathbf{w}}}^{\prime })-P(\overline{\mathbf{w}})\Vert \le \frac{\gamma }{{\sigma }_{\mathcal{M}}}(\delta +\eta ),$$

$$d_p([{\overline{\mathbf{w}}}^{\prime }],[\overline{\mathbf{w}}])\le 7(HS{)}^{\frac{1}{p}}\min \left(\Vert P(\overline{\mathbf{w}}){\Vert }_{\infty }^{\frac{1}{H}-1},\Vert P({\overline{\mathbf{w}}}^{\prime }){\Vert }_{\infty }^{\frac{1}{H}-1}\right)\frac{\gamma }{{\sigma }_{\mathcal{M}}}(\delta +\eta ).$$

$$\Vert {\mathcal{A}}_{{\overline{\mathcal{S}}}^{\prime }\cup \overline{\mathcal{S}}}(P({\overline{\mathbf{w}}}^{\prime })-P(\overline{\mathbf{w}}))\Vert$$

$$P({\overline{\mathbf{w}}}^{\prime })-P(\overline{\mathbf{w}})=T+T^{\prime },$$

Full text

The analysis is based on the recently proposed Tensorial Lifting. The particularity of this paper is to consider a sparsity prior. This leads to a better stability constant. As an illustration, we detail the analysis and provide sharp stability guarantees for convolutional linear network under sparsity prior. In this analysis, we distinguish the role of the network architecture and the sample input. This highlights the requirements on the data in connection to parameter stability.

1 Introduction

1.1 The stability property

Artificial neural networks have improved the state of the art and continue to improve it in a large number of applications in science and technology. Their empirical success far exceeds the understanding of their theoretical properties. In particular, despite the very significant efforts of a very active research community, some behaviors remain partially understood: Why do optimization algorithms find good solutions? Why do over-parameterized neural networks retain good generalization properties? What classes of functions can be approximated by neural networks? With which minimal network architecture?

The work presented in this paper is of a theoretical nature and focuses on a stability property for the parameters leading to a low objective function. The statements are for a regression problem. To explain this stability property in a simplified context, we consider a parameterized family of functions $f_{\mathbf{w}}$ (e.g. neural networks), the parameter being $\mathbf{w}$ ; the parameter space is equipped with a metric111To be accurate, the metric is defined between equivalence classes reflecting invariance properties of the family $f$. For instance, in the case of neural networks, we would like to consider weight rescaling and/or neurons re-arrangement. $d$; we consider a sample $(x_{i},y_{i})_{i=1..n}$ of size $n\in\mathbb{N}$. The stability statement then takes the following form.

Informal theorem 1

Stability Guarantee**

If a certain condition on the family $f$ and the sample is satisfied then we have the following stability property:

There exists $C>0$ such that for $\eta$ sufficiently small: For any $\mathbf{w}$ and $\mathbf{w}^{\prime}$ such that

[TABLE]

we have

[TABLE]

Notice first that the above informal theorem provides a sufficient condition guaranteeing the stability property. Subsequently, depending on the nature of the network under consideration, necessary and sufficient conditions or necessary conditions will be stated. The interest of the stability property is that it guarantees:

• •

**Feature stability and interpretability: ** When the optimal $\mathbf{w}$ is stable, the features in the latent spaces and the output of the network are stably defined in the sense that the parameters $\mathbf{w}$ and $\mathbf{w}^{\prime}$ for which $\eta$ is small define similar features and output. The features and the output only depend on the value of

[TABLE]

and do not depend on the algorithm used to find $\mathbf{w}$. In particular, they do not depend on its initialization, the numerical parameters, the order of the samples in the stochastic algorithm, the numerical tricks etc The parameter $\mathbf{w}$ and therefore the function $f_{\mathbf{w}}$ only depends on $f$ (i.e.: the network architecture, for neural networks) and the sample $(x_{i},y_{i})_{i=1..n}$. For neural networks, this is a strong guaranty when interpreting the influence of the features on the output.

• •

**Stable recovery: ** If we make the additional assumption that the data are generated from the family $f$ for an ideal parameter $\mathbf{w}$ (up to an accuracy smaller than $\eta$), then the stability guaranty ensures that any parameter $\mathbf{w}^{\prime}$ for which

[TABLE]

is sufficiently small is close to the ideal $\mathbf{w}$.

The above additional assumption can be provided by approximation theory statement. This is, for instance, the usual argument in compressed-sensing [Elad(2010)]. When solving a linear inverse problem under sparsity constraints, the sparsity hypothesis is not so restrictive because many signals/images classes are compressible. We can expect the same phenomenon to happen for neural networks for which such statements are often referred to as “expressivity” or “expressive power”. For instance, we can expect to have such guaranties when the neural network approximates a smooth function [Boölcskei et al.(2019)Boölcskei, Grohs, Kutyniok, and Petersen, Gühring et al.(2019)Gühring, Kutyniok, and Petersen, Gribonval et al.(2019)Gribonval, Kutyniok, Nielsen, and Voigtlaender].

1.2 Existing results on the stable recovery

Establishing stable recovery guarantees for neural networks is a difficult subject which has not been addressed very often. The subject remains largely unexplored. To the best of our knowledge conditions guaranteeing the stability property for neural networks have been established in [Arora et al.(2014)Arora, Bhaskara, Ge, and Ma, Brutzkus and Globerson(2017), Li and Yuan(2017), Sedghi and Anandkumar(2014), Zhong et al.(2017)Zhong, Song, Jain, Bartlett, and Dhillon, Malgouyres and Landsberg(2016), Malgouyres and Landsberg(2019)]. A negative statement, exhibiting an unstable configuration when the weights go to infinity is given in [Petersen et al.(2019)Petersen, Voigtlaender, and Raslan].

Among them, [Brutzkus and Globerson(2017), Li and Yuan(2017), Zhong et al.(2017)Zhong, Song, Jain, Bartlett, and Dhillon] consider a family of networks with one hidden layer. The article [Sedghi and Anandkumar(2014)] focuses on the recovery of the parameters defining one layer in a arbitrarily deep networks. The articles [Arora et al.(2014)Arora, Bhaskara, Ge, and Ma, Malgouyres and Landsberg(2016), Malgouyres and Landsberg(2019)] consider networks without depth limitation.

In [Brutzkus and Globerson(2017)], the authors consider the minimization of the population risk. The input is assumed Gaussian and the output is generated by a network involving one linear layer followed by ReLU and a mean. The number of intermediate nodes is smaller than the input size. They provide conditions guaranteeing that, with high probability, a randomly initialized gradient descent algorithm converges to the true parameters. The authors of [Li and Yuan(2017)] consider a framework similar to [Brutzkus and Globerson(2017)]. They show that the stochastic gradient descent converges to the true solution. In [Zhong et al.(2017)Zhong, Song, Jain, Bartlett, and Dhillon], the authors consider a non-linear layer followed by a linear layer. The size of the intermediate layer is smaller than the size of the input and the size of the output is $1$. They prove that the gradient algorithm minimizing the empirical risk converges to the true parameters, for the particular initialization described in the article.

The authors of [Sedghi and Anandkumar(2014)] consider a feed-forward neural network and show that, if the input is Gaussian or its distribution is known, a method based on moments and sparse dictionary learning can retrieve the parameters defining the first linear transform. Nothing is said about the stability or the estimation of the other transformations.

The authors of [Arora et al.(2014)Arora, Bhaskara, Ge, and Ma] consider deep feed-forward networks which are very sparse and randomly generated. They show that they can be learned with high probability one layer after another. However, very sparse and randomly generated networks are not used in practice and one might want to study more versatile structures.

The article [Malgouyres and Landsberg(2016)] studies deep structured linear networks and uses the same tensorial lifting we use here. This result has been extended in [Malgouyres and Landsberg(2019)], where necessary and sufficient conditions of stable recovery have been established for a general constraint on the parameters defining the network. In the present article, we specialize the analysis to the sparsity constraint. We also obtain necessary and sufficient conditions of stable recovery. However, we obtain a better stability constant (the constant $C$ in Informal Theorem 1). The difference is of the same nature as when the smallest singular value is replaced by a lower RIP constant in compressed sensing [Elad(2010)]. Moreover, in the analysis dedicated to convolutional linear networks, we separate the hypotheses on the data $(x_{i})_{i=1..n}$ and the network architecture. This highlights the importance of having a full row rank $X$, where $X$ is the concatenation of the data $(x_{i})_{i=1..n}$, and shows the role of the smallest singular value of $X$ in this context. These are the two main contributions of the paper.

Finally, denoting $H$ the number of factors/layers, the approach developed in this paper extends to $H\geq 3$ existing compressed sensing results for $H\leq 2$. In particular, when $H=1$, the considered problems boils down to a compressed sensing problem [Elad(2010)]. When $H=2$ and when extended to other constraints on the parameters $\mathbf{w}$, the statements apply to already studied problems such as: low rank approximation [Candes et al.(2013)Candes, Strohmer, and Voroninski], Non-negative matrix factorization [Lee and Seung(1999), Donoho and Stodden(2003), Laurberg et al.(2008)Laurberg, Christensen, Plumbley, Hansen, and Jensen, Arora et al.(2012)Arora, Ge, Kannan, and Moitra], dictionary learning [Jenatton et al.(2012)Jenatton, Gribonval, and Bach], phase retrieval [Candes et al.(2013)Candes, Strohmer, and Voroninski], blind deconvolution [Ahmed et al.(2014)Ahmed, Recht, and Romberg, Choudhary and Mitra(2014), Li et al.(2016)Li, Ling, Strohmer, and Wei]. Most of these papers use the same lifting property we are using. They further propose to convexify the problem. A more general bilinear framework is considered in [Choudhary and Mitra(2014)].

1.3 The considered sparse networks

As in [Malgouyres and Landsberg(2019)], we consider structured linear networks. The layers can be convolutional or feedforward. The network has at least one hidden layer and can be deep. The network is not biased. We give in this section all the notations on networks.

Throughout the paper, we consider $H\geq 2$, $S\geq 2$, $m_{0}\ldots m_{H}\in\mathbb{N}$ and write $m_{H}=m$. We consider a network and assume its architecture fixed. It has $H-1$ hidden layers. The layer [math] corresponds to the inputs, the layer $H$ to the output. The hidden layers correspond to the indexes $1,\cdots,H-1$. For $h\in\{0,\cdots,H\}$, $m_{h}$ is the size of the layer $h$. We assume that the whole network is parameterized by an element of $\mathbb{R}^{S\times H}$, say $\mathbf{w}\in\mathbb{R}^{S\times H}$. The architecture of the network is defined by linear mappings

[TABLE]

for $h\in\{1,\cdots,H\}$. For all $h\in\{1,\cdots,H\}$, the linear part of the transformation that maps the content of the layer $h-1$ to the layer $h$ is parameterized by $\mathbf{w}_{h}\in\mathbb{R}^{S}$ and is defined by $M_{h}(\mathbf{w}_{h})$. Modeling the architecture of the network with the operators $M_{h}$, we can consider many kind of networks. Indeed, depending on the operators $M_{h}$, the network can include feedforward layers, convolutional layers and other structured layers tailored to particular structures in the data. The layers might not be fully connected.

The mapping from $\mathbb{R}^{m_{0}}$ to $\mathbb{R}^{m}$ defined by the network is called the prediction and it is defined for any $x\in\mathbb{R}^{m_{0}}$ by

[TABLE]

We use the same notation $f_{\mathbf{w}}$ when applying $f_{\mathbf{w}}$ to every column of $X\in\mathbb{R}^{m_{0}\times n}$ and concatenating the results in a matrix in $\mathbb{R}^{m\times n}$. The abuse of notation is not ambiguous, once in context.

Again, the considered networks do not involve activation functions and biases. However, as indicated in [Malgouyres and Landsberg(2019)], the action of the ReLU activation function multiplies the content of any neuron by an element of $\{0,1\}$. The choice of the element depends on $\mathbf{w}$ and $x$. However, considering $x$ fixed, since $\{0,1\}$ is finite, there is a finite set of possibilities for the action of the ReLU activation function. Said differently, there is a finite number of possibilities for the choice of the neurons that are kept. Therefore, there exists a partition of $\mathbb{R}^{S\times H}$ such that, on every piece of the partition, the action of ReLU is constant. Therefore, on every piece of the partition, the network is a structured linear network as studied in the present paper. Notice moreover that the analysis in [Choromanska et al.(2015a)Choromanska, Henaff, Mathieu, Arous, and LeCun, Choromanska et al.(2015b)Choromanska, LeCun, and Arous] take the expectation of the action of ReLU networks (under an un-realistic independence hypothesis) and obtain a structured linear network. Beside, structured linear network are significantly more general than the deep linear networks that are often considered (see, among other, [Baldi and Hornik(1989), Kawaguchi(2016)]).

Throughout the paper, we consider a family of possible supports ${{\mathcal{M}}}\subset{\mathcal{P}}(\{1,\cdots,S\}^{H})$, where ${\mathcal{P}}(\{1,\cdots,S\}^{H})$ denotes the set of all possible supports (the parts of $\{1,\cdots,S\}^{H}$). A classical example is ${{\mathcal{M}}}=\{\mathcal{S}|\forall h=1..h,|\mathcal{S}_{h}|\leq S^{\prime}\}$, for a given $S^{\prime}\leq S$. We constrain the parameter $\mathbf{w}$ to satisfy a sparsity constraint of the form : there exists ${\mathcal{S}}=(\mathcal{S}_{h})_{h=1..H}\in{{\mathcal{M}}}$ such that

[TABLE]

(i.e.: $\forall h$, $\operatorname{supp}\left({\mathbf{w}}_{h}\right)\subset{\mathcal{S}}_{h}$). Specializing the analysis to sparsity constraints is one of the main differences between this paper and [Malgouyres and Landsberg(2016), Malgouyres and Landsberg(2019)]. Sparse networks have been considered in many contexts [Ranzato et al.(2007)Ranzato, Poultney, Chopra, and Cun, Ranzato et al.(2008)Ranzato, Boureau, and Cun, Lee et al.(2008)Lee, Ekanadham, and Ng, Srinivas et al.(2017)Srinivas, Subramanya, and Venkatesh Babu, Louizos et al.(2018)Louizos, Welling, and Kingma, Zhang et al.(2016)Zhang, Du, Zhang, Lan, Liu, Li, Guo, Chen, and Chen]; sparse convolutional neural networks have also been considered [Liu et al.(2015)Liu, Wang, Foroosh, Tappen, and Pensky].

1.4 The solutions of the problem

We assume that data are collected in the columns of matrices $X\in\mathbb{R}^{m_{0}\times n}$ and $Y\in\mathbb{R}^{m_{H}\times n}$.

To establish the stability property, we consider throughout the paper $\overline{\mathcal{S}}$ and ${\overline{\mathcal{S}}}^{\prime}\in{{\mathcal{M}}}$, $\overline{\mathbf{w}}$ and $\overline{\mathbf{w}}^{\prime}\in\mathbb{R}^{S\times H}$ such that

[TABLE]

and for which

[TABLE]

are small. Generic parameters are denoted without the over-line: $\mathcal{S}$, $\mathcal{S}^{\prime}$, $\mathbf{w}$, $\mathbf{w}^{\prime}$ etc

We want to establish a condition guaranteeing that, up to a multiplicative constant, the distance between such $\overline{\mathbf{w}}$ and $\overline{\mathbf{w}}^{\prime}$ is upper-bound by $\delta+\eta$. As already said, using a true distance would be too restrictive and the true statements involve a distance between equivalence classes of parameters.

2 Notations and preliminaries on Tensorial Lifting

Set $[\![H]\!]=\{1,\ldots,H\}$ and $\mathbb{R}^{S\times H}_{*}=\{\mathbf{w}\in\mathbb{R}^{S\times H}|\forall h=1..H,\|\mathbf{w}_{h}\|\neq 0\}$, where we remind that $\mathbf{w}_{h}\in\mathbb{R}^{S}$ contains the parameters defining the transform between layers $h-1$ and $h$. Define an equivalence relation in $\mathbb{R}^{S\times H}_{*}$: for any $\mathbf{w}$, $\mathbf{v}\in\mathbb{R}^{S\times H}$, $\mathbf{w}\sim\mathbf{v}$ if and only if there exists $(\lambda_{h})_{h=1..H}\in\mathbb{R}^{H}$ such that

[TABLE]

Denote the equivalence class of $\mathbf{w}\in\mathbb{R}^{S\times H}_{*}$ by $[\mathbf{w}]$. For any $p\in[1,\infty]$, we denote the usual $\ell^{p}$ norm by $\|.\|_{p}$ and define the mapping $d_{p}:\left((\mathbb{R}^{S\times H}_{*}/\sim)\times(\mathbb{R}^{S\times H}_{*}/\sim)\right)\rightarrow\mathbb{R}$ by

[TABLE]

where

[TABLE]

It is proved in [Malgouyres and Landsberg(2019)] that $d_{p}$ is a metric on $\mathbb{R}^{S\times H}_{*}/\sim$.

The real valued tensors of order $H$ whose axes are of size $S$ are denoted by $T\in\mathbb{R}^{S\times\ldots\times S}$. The space of tensors is abbreviated $\mathbb{R}^{S^{H}}$. We say that a tensor $T\in\mathbb{R}^{S^{H}}$ is of rank $1$ if and only if there exists a collection of vectors $\mathbf{w}\in\mathbb{R}^{S\times H}$ such that, for any $\mathbf{i}=(i_{1},\ldots,i_{H})\in[\![S]\!]^{H}$,

[TABLE]

The set of all the tensors of rank less than $1$ is denoted by $\Sigma_{1}$. We denote $\Sigma_{2}=\Sigma_{1}+\Sigma_{1}$. Moreover, we parameterize $\Sigma_{1}\subset\mathbb{R}^{S^{H}}$ using the Segre embedding

[TABLE]

As stated in the next two theorems, we can control the distortion of the distance induced by $P$ and its ‘inverse’.

Theorem 1

Stability of $[\mathbf{w}]$ from $P(\mathbf{w})$, see [Malgouyres and Landsberg(2019)]**

Let $\mathbf{w}$ and $\mathbf{w}^{\prime}\in\mathbb{R}^{S\times H}_{*}$ be such that $\|P(\mathbf{w}^{\prime})-P(\mathbf{w})\|_{\infty}\leq\frac{1}{2}~{}\max\left(\|P(\mathbf{w})\|_{\infty},\|P(\mathbf{w}^{\prime})\|_{\infty}\right)$. For all $p,q\in[1,\infty]$,

[TABLE]

Theorem 2

‘Lipschitz’ continuity of $P$, see [Malgouyres and Landsberg(2019)]**

We have for any $q\in[1,\infty]$ and any $\mathbf{w}$ and $\mathbf{w}^{\prime}\in\mathbb{R}^{S\times H}_{*}$,

[TABLE]

The Tensorial Lifting (see [Malgouyres and Landsberg(2019)]) states that for any $M_{1}$, …, $M_{H}$ and any $X$ there exists a unique linear map

[TABLE]

such that for all $\mathbf{w}\in\mathbb{R}^{S\times H}$

[TABLE]

The intuition leading to this equality is that every entry in $M_{H}(\mathbf{w}_{H})\cdots M_{1}(\mathbf{w}_{1})X$ is a multivariate polynomial whose variables are in $\mathbf{w}$. Moreover, every monomial of the polynomials is of the form $a_{\mathbf{i}}P(\mathbf{w})_{\mathbf{i}}$ for $\mathbf{i}\in[\![S]\!]^{H}$, where $a_{\mathbf{i}}$ is a coefficient which depends on $M_{1}$, …, $M_{H}$ and $X$. The Tensorial Lifting expresses any deep structured linear network using the Segre Embedding and a linear operator $\mathcal{A}$. The Segre embedding is non-linear and might seem difficult to deal with at the first sight, but it is always the same whatever the network architecture, the sparsity pattern, the action of the ReLU activation function…These constituents of the problem only influence the lifting linear operator $\mathcal{A}$.

In the next section, we study what properties of $\mathcal{A}$ are required to obtain the stable recovery. In Section 4, we study these properties when $\mathcal{A}$ corresponds to a sparse convolutional linear network.

3 General conditions for the stable recovery under sparsity constraint

From now on, the analysis differs from the one presented in [Malgouyres and Landsberg(2019)]. It is dedicated to models that enforce sparsity. In this particular situation, we can indeed have a different view of the geometry of the problem. In order to describe it, we first establish some notation.

We define a support by $\mathcal{S}=(\mathcal{S}_{h})_{h=1..H}$, with $\mathcal{S}_{h}\subset[\![S]\!]$, and remind that we denote the set of all supports by ${\mathcal{P}}([\![S]\!]^{H})$ (the parts of $[\![S]\!]^{H}$). For a given support $\mathcal{S}\in{\mathcal{P}}([\![S]\!]^{H})$, we denote

[TABLE]

(i.e., for all $h$, $\operatorname{supp}\left(\mathbf{w}_{h}\right)\subset\mathcal{S}_{h}$) and

[TABLE]

We also denote by ${\mathbf{P}}_{\mathcal{S}}$ the orthogonal projection from $\mathbb{R}^{S^{H}}$ onto $\mathbb{R}^{S^{H}}_{\mathcal{S}}$. It has the closed-form expression: for all $T\in\mathbb{R}^{S^{H}}$ and all $\mathbf{i}\in[\![S]\!]^{H}$

[TABLE]

We consider different operators and define for any $\mathcal{S}\in{\mathcal{P}}([\![S]\!]^{H})$

[TABLE]

We will use later on that for any $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{\mathcal{M}}$ and for any $\mathbf{w}\in\mathbb{R}^{S\times H}_{\mathcal{S}}$, or any $\mathbf{w}\in\mathbb{R}^{S\times H}_{\mathcal{S}^{\prime}}$, or any $\mathbf{w}\in\mathbb{R}^{S\times H}_{\mathcal{S}\cup\mathcal{S}^{\prime}}$, we have

[TABLE]

The introduction of the different operators $\mathcal{A}_{\mathcal{S}}$ leads to an analysis different from the one conducted in [Malgouyres and Landsberg(2019)]. Instead of considering the intersection of one linear space with a subset of $\Sigma_{2}$ (as in [Malgouyres and Landsberg(2019)]), we consider the intersection of many linear sets (the kernels of the operator $\mathcal{A}_{\mathcal{S}}$) with $\Sigma_{1}$.

The following property will turn out to be necessary and sufficient to guarantee the stable recovery property.

Definition 1

Sparse-Deep-Null Space Property**

Let $\gamma\geq 1$ and $\rho>0$, we say that $\mathcal{A}$ satisfies the sparse-deep-Null Space Property (sparse-deep-NSP ) with constants $(\gamma,\rho)$ for ${\mathcal{M}}$ if and only if for all $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{\mathcal{M}}$, any $T\in P(\mathbb{R}^{S\times H}_{\mathcal{S}})+P(\mathbb{R}^{S\times H}_{\mathcal{S}^{\prime}})$ satisfying $\|\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}T\|\leq\rho$ and any $T^{\prime}\in\operatorname{Ker}\left(\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\right)$, we have

[TABLE]

Geometrically, the sparse-deep-NSP does not hold when ${\mathbf{P}}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\operatorname{Ker}\left(\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\right)$ intersects $P(\mathbb{R}^{S\times H}_{\mathcal{S}})+P(\mathbb{R}^{S\times H}_{\mathcal{S}^{\prime}})$ away from the origin or tangentially at [math]. It holds when the two sets intersect ”transversally” at [math]. Despite an apparent abstract nature, we will be able to characterize precisely when the lifting operator corresponding to a convolutional linear network satisfies the sparse-deep-NSP (see Section 4). We will also be able to calculate the constants $(\gamma,\rho)$.

Proposition 1

**Sufficient condition for sparse-deep-NSP ****

If $\operatorname{Ker}\left(\mathcal{A}\right)\cap\mathbb{R}^{S^{H}}_{\mathcal{S}\cup\mathcal{S}^{\prime}}=\{0\}$, for all $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{\mathcal{M}}$, then $\mathcal{A}$ satisfies the sparse-deep-NSP with constants $(\gamma,\rho)=(1,+\infty)$ for ${\mathcal{M}}$.

\proof

In order to prove the proposition, let us consider $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{\mathcal{M}}$, $T^{\prime}\in\operatorname{Ker}\left(\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\right)$. We have $\mathcal{A}{\mathbf{P}}_{\mathcal{S}\cup\mathcal{S}^{\prime}}T^{\prime}=0$ and therefore ${\mathbf{P}}_{\mathcal{S}\cup\mathcal{S}^{\prime}}T^{\prime}\in\operatorname{Ker}\left(\mathcal{A}\right)$. Moreover, by definition, ${\mathbf{P}}_{\mathcal{S}\cup\mathcal{S}^{\prime}}T^{\prime}\in\mathbb{R}^{S^{H}}_{\mathcal{S}\cup\mathcal{S}^{\prime}}$. Therefore, applying the hypothesis of the proposition, we obtain ${\mathbf{P}}_{\mathcal{S}\cup\mathcal{S}^{\prime}}T^{\prime}=0$ and (11) holds for any $T$, when $\gamma=1$. Therefore, $\mathcal{A}$ satisfies the sparse-deep-NSP with constants $(\gamma,\rho)=(1,+\infty)$ for ${\mathcal{M}}$.

\endproof

If $[\![S]\!]^{H}\in{\mathcal{M}}$, the condition becomes $\operatorname{Ker}\left(\mathcal{A}\right)=\{0\}$, which is sufficient but obviously not necessary for the sparse-deep-NSP to hold. However, when ${\mathcal{M}}$ truly imposes sparsity, the condition $\operatorname{Ker}\left(\mathcal{A}\right)\cap\mathbb{R}^{S^{H}}_{\mathcal{S}\cup\mathcal{S}^{\prime}}=\{0\}$ says that the elements of $\operatorname{Ker}\left(\mathcal{A}\right)$ shall not be sparse in some (tensorial) way. This nicely generalizes the case $H=1$.

Definition 2

Deep-lower-RIP constant**

There exists a constant $\sigma_{{\mathcal{M}}}>0$ such that for any $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{\mathcal{M}}$ and any $T$ in the orthogonal complement of $\operatorname{Ker}\left(\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\right)$

[TABLE]

We call $\sigma_{{\mathcal{M}}}$ a Deep-lower-RIP constant of $\mathcal{A}$ with regard to ${\mathcal{M}}$.

\proof

The existence of $\sigma_{{\mathcal{M}}}$ is a straightforward consequence of the fact that the restriction of $\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}$ on the orthogonal complement of $\operatorname{Ker}\left(\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\right)$ is injective. We therefore have for all $T$ in the orthogonal complement of $\operatorname{Ker}\left(\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\right)$

[TABLE]

where $\sigma_{\mathcal{S}\cup\mathcal{S}^{\prime}}>0$ is the smallest non-zero singular value of $\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}$. The last inequality holds because ${\mathbf{P}}_{\mathcal{S}\cup\mathcal{S}^{\prime}}$ is a contraction.

We obtain the existence of $\sigma_{{\mathcal{M}}}$ by taking the minimum of the constants $\sigma_{\mathcal{S}\cup\mathcal{S}^{\prime}}$ over the finite family of $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{\mathcal{M}}$. \endproof

Theorem 3

Sufficient condition of stable recovery for structured linear networks**

Consider a structured linear network defined by $M_{1}$, …, $M_{H}$, sparsity constraints defined by a family of possible supports ${\mathcal{M}}$, data $X$ and $Y$ and the operator $\mathcal{A}$ satisfying (7).

Assume $\mathcal{A}$ satisfies the sparse-deep-NSP with the constants $\gamma\geq 1$, $\rho>0$ for ${\mathcal{M}}$. For any $\overline{\mathcal{S}}\in{\mathcal{M}}$, $\overline{\mathbf{w}}\in\mathbb{R}^{S\times H}_{\overline{\mathcal{S}}}$ and $\overline{\mathcal{S}}^{\prime}\in{\mathcal{M}}$, $\overline{\mathbf{w}}^{\prime}\in\mathbb{R}^{S\times H}_{\overline{\mathcal{S}}^{\prime}}$ as in (2) with $\eta+\delta\leq\rho$, we have

[TABLE]

where $\sigma_{{\mathcal{M}}}$ is the Deep-lower-RIP constant of $\mathcal{A}$ with regard to ${\mathcal{M}}$.

Moreover, if $\frac{\gamma}{\sigma_{{\mathcal{M}}}}~{}(\delta+\eta)\leq\frac{1}{2}~{}\max\left(\|P(\overline{\mathbf{w}}^{\prime})\|_{\infty},\|P(\overline{\mathbf{w}})\|_{\infty}\right)$, then

[TABLE]

Proof 3.1**.**

Because $\mathcal{A}_{{\overline{\mathcal{S}}}^{\prime}\cup\overline{\mathcal{S}}}$ is linear and then because $\overline{\mathbf{w}}\in\mathbb{R}^{S\times H}_{\overline{\mathcal{S}}}$ and $\overline{\mathbf{w}}^{\prime}\in\mathbb{R}^{S\times H}_{\overline{\mathcal{S}}^{\prime}}$, using (3), we have

[TABLE]

If we further decompose (the decomposition is unique)

[TABLE]

where $T^{\prime}\in\operatorname{Ker}\left(\mathcal{A}_{{\overline{\mathcal{S}}}^{\prime}\cup\overline{\mathcal{S}}}\right)$ and $T$ is orthogonal to $\operatorname{Ker}\left(\mathcal{A}_{{\overline{\mathcal{S}}}^{\prime}\cup\overline{\mathcal{S}}}\right)$, we have

[TABLE]

where $\sigma_{{\mathcal{M}}}$ is the Deep-lower-RIP constant of $\mathcal{A}$ with regard to ${\mathcal{M}}$. Combining with (13), we get

[TABLE]

*Combining this inequality with ${\mathbf{P}}_{{\overline{\mathcal{S}}}^{\prime}\cup\overline{\mathcal{S}}}P({\overline{\mathbf{w}}}^{\prime})=P({\overline{\mathbf{w}}}^{\prime})$, ${\mathbf{P}}_{{\overline{\mathcal{S}}}^{\prime}\cup\overline{\mathcal{S}}}P(\overline{\mathbf{w}})=P(\overline{\mathbf{w}})$ and (14), we obtain *

[TABLE]

Combining the latter inequality with the hypotheses:$\mathcal{A}$ satisfies the sparse-deep-NSP with constants $(\gamma,\rho)$ for ${\mathcal{M}}$ and $\delta+\eta\leq\rho$; we have

[TABLE]

When $\delta+\eta$ satisfy the condition in the theorem, we can apply Theorem 1 and obtain the last inequality.

Theorem 3 differs from the analogous theorem in [Malgouyres and Landsberg(2019)]. In particular, it is dedicated to sparsity constraints. The constant of the upper bound is different. We replace the smallest non-zero singular value of an operator by the min, over a finite number of linear space, of the smallest non-zero singular value of the restriction of the operator on the linear space (see Definition 2 and its proof). This is the usual role of the lower-RIP constant in compressed sensing [Elad(2010)], hence the name Deep-lower-RIP.

One might again ask whether the condition “$\mathcal{A}$ satisfies the sparse-deep-NSP ”  is sharp or not. As stated in the following theorem, the answer is affirmative.

Theorem 4**.**

Necessary condition for stable recovery for structured linear networks**

Consider a structured linear network defined by $M_{1}$, …, $M_{H}$, , sparsity constraints defined by a family of possible supports ${\mathcal{M}}$, data $X$ and $Y$ and the operator $\mathcal{A}$ satisfying (7).

Assume the stability property holds: There exists $C$ and $\delta>0$ such that for any $\overline{\mathcal{S}}\in{\mathcal{M}}$ and any $\overline{\mathbf{w}}\in\mathbb{R}^{S\times H}_{\overline{\mathcal{S}}}$, any $Y=\mathcal{A}P(\overline{\mathbf{w}})+e$, with $\|e\|\leq\delta$, and any ${\overline{\mathcal{S}}}^{\prime}\in{\mathcal{M}}$ and ${\overline{\mathbf{w}}}^{\prime}\in\mathbb{R}^{S\times H}_{{\overline{\mathcal{S}}}^{\prime}}$ such that

[TABLE]

we have

[TABLE]

Then, $\mathcal{A}$ satisfies the sparse-deep-NSP with constants

[TABLE]

for ${\mathcal{M}}$, where $\sigma_{max}$ is the spectral radius of $\mathcal{A}$.

Proof 3.2**.**

Let $\overline{\mathcal{S}}$ and $\overline{\mathcal{S}}^{\prime}\in{\mathcal{M}}$. Let $\overline{\mathbf{w}}\in\mathbb{R}^{S\times H}_{\overline{\mathcal{S}}}$ and $\overline{\mathbf{w}}^{\prime}\in\mathbb{R}^{S\times H}_{\overline{\mathcal{S}}^{\prime}}$ be such that $\|\mathcal{A}\left(P(\overline{\mathbf{w}})-P(\overline{\mathbf{w}}^{\prime})\right)\|\leq\delta$. We have, using (3),

[TABLE]

Throughout the proof, we also consider $T^{\prime}\in\operatorname{Ker}\left(\mathcal{A}_{\overline{\mathcal{S}}\cup\overline{\mathcal{S}}^{\prime}}\right)$. We assume that $\|P(\overline{\mathbf{w}})\|_{\infty}\leq\|P(\overline{\mathbf{w}}^{\prime})\|_{\infty}$. When it is not the case, the proof is analogue. We denote

[TABLE]

We have $Y=\mathcal{A}P(\overline{\mathbf{w}})+e$ with $\|e\|\leq\delta$. Moreover, since $\overline{\mathbf{w}}\in\mathbb{R}^{S\times H}_{\overline{\mathcal{S}}}$, $\|e\|\leq\delta$ and since we obviously have $\|\mathcal{A}P({\overline{\mathbf{w}}}^{\prime})-Y\|\leq\|e\|$, the assumption that the stability property holds guaranties

[TABLE]

Using (3) and the fact that $e=\mathcal{A}_{\overline{\mathcal{S}}\cup\overline{\mathcal{S}}^{\prime}}(P(\overline{\mathbf{w}})-P(\overline{\mathbf{w}}^{\prime}))$, for any $T^{\prime}\in\operatorname{Ker}\left(\mathcal{A}_{\overline{\mathcal{S}}\cup\overline{\mathcal{S}}^{\prime}}\right)$

[TABLE]

where $\sigma_{max}$ is the spectral radius of $\mathcal{A}$. Therefore,

[TABLE]

Finally, using Theorem 2 and the fact that $\|P(\overline{\mathbf{w}})\|_{\infty}\leq\|P(\overline{\mathbf{w}}^{\prime})\|_{\infty}$, we obtain

[TABLE]

for $\gamma=CS^{\frac{H-1}{2}}\sqrt{H}~{}\sigma_{max}~{}$.

Summarizing, we conclude that under the hypothesis of the theorem: For any $\overline{\mathcal{S}}$ and $\overline{\mathcal{S}}^{\prime}\in{\mathcal{M}}$ and any $T\in P(\mathbb{R}^{S\times H}_{\overline{\mathcal{S}}})+P(\mathbb{R}^{S\times H}_{\overline{\mathcal{S}}^{\prime}})$ (above $P(\overline{\mathbf{w}})-P(\overline{\mathbf{w}}^{\prime})$ has the role of $T$) such that $\|\mathcal{A}T\|=\|\mathcal{A}_{\overline{\mathcal{S}}\cup\overline{\mathcal{S}}^{\prime}}T\|\leq\delta$, we have for any $T^{\prime}\in\operatorname{Ker}\left(\mathcal{A}_{\overline{\mathcal{S}}\cup\overline{\mathcal{S}}^{\prime}}\right)$

[TABLE]

In words, $\mathcal{A}$ satisfies the sparse-deep-NSP for ${\mathcal{M}}$ with the constants of Theorem 4.

4 Application to convolutional linear network under sparsity prior

We consider a sparse convolutional linear network as depicted in Figure 1. Formally, the considered convolutional linear network is defined from a rooted directed acyclic graph $\mathcal{G}(\mathcal{E},\mathcal{N})$ composed of nodes $\mathcal{N}$ and edges $\mathcal{E}$. Each edge connects two nodes. The root of the graph is denoted by $r$ (it contains the output signal) and the set containing all its leaves is denoted by $\mathcal{F}$ (the leaves contain the input signal). We denote by ${\mathcal{P}}$ the set of all paths connecting the leaves and the root. We assume, without loss of generality, that the length of any path between a leaf and the root is independent of the considered path and equal to $H\geq 0$. We also assume that, for any edge $e\in\mathcal{E}$, the length of the paths separating $e$ and any leaf is constant. This length is called the depth of $e$. For any $h=1..H$, we denote the set containing all the edges of depth $h$, by $\mathcal{E}(h)$.

Moreover, to any edge $e$ is attached a convolution kernel of maximal support $\mathcal{S}_{e}\subset[\![N]\!]$. We assume (without loss of generality) that $\sum_{e\in\mathcal{E}(h)}|\mathcal{S}_{e}|$ is independent of $h$ ($|\mathcal{S}_{e}|$ denotes the cardinality of $\mathcal{S}_{e}$). We take

[TABLE]

For any edge $e$, we consider the mapping $\mathcal{T}_{e}:\mathbb{R}^{S}\longrightarrow\mathbb{R}^{N}$ that maps any $w\in\mathbb{R}^{S}$ into the convolution kernel $\mathcal{T}_{e}(w)\in\mathbb{R}^{N}$, attached to the edge $e$, whose support is $\mathcal{S}_{e}$. As in the previous section, we assume a sparsity constraint and will only consider a family ${{\mathcal{M}}}$ of possible supports $\mathcal{S}\subset[\![S]\!]^{H}$.

At each $h$, the convolutional linear network computes, for all $e\in\mathcal{E}(h)$, the convolution between the signal at the origin of $e$; then, it attaches to any ending node the sum of all the convolutions arriving at that node. Examples of such convolutional linear networks includes wavelets, wavelet packets [Mallat(1998)] or the fast transforms optimized in [Chabiron et al.(2014)Chabiron, Malgouyres, Tourneret, and Dobigeon, Chabiron et al.(2016)Chabiron, Malgouyres, Wendt, and Tourneret]. It is the usual convolutional neural network, without bias, in which the activation function is the identity and the supports are potentially scattered and not fixed. It is clear that the operation performed between any pair of consecutive layers depends linearly on parameters $w\in\mathbb{R}^{S}$. The convolutional linear network therefore depends on parameters $\mathbf{w}\in\mathbb{R}^{S\times H}$ and its prediction takes the form

[TABLE]

where the operators $M_{h}$ satisfy the hypothesis of the present paper and $x^{|\mathcal{F}|}=M_{0}x$ where $M_{0}$ concatenates vertically $|\mathcal{F}|$ identity matrix of size $N\times N$:

[TABLE]

Given a sample $(x_{i},y_{i})_{i=1..n}\in(\mathbb{R}^{N}\times\mathbb{R}^{N})^{n}$ and reminding that $X$ is the horizontal concatenation of the column vectors $x_{i}$, we also denote $X^{|\mathcal{F}|}=M_{0}X\in\mathbb{R}^{N|\mathcal{F}|\times n}$.

Given $X^{|\mathcal{F}|}$ and a network architecture, this section applies the results of the preceding sections in order to identify sharp conditions guaranteeing that, for any supports $\overline{\mathcal{S}}$ and $\overline{\mathcal{S}}^{\prime}\in{{\mathcal{M}}}$, any parameters $\overline{\mathbf{w}}$ and $\overline{\mathbf{w}}^{\prime}\in\mathbb{R}^{S\times H}$ satisfying $\operatorname{supp}\left(\overline{\mathbf{w}}\right)\subset\overline{\mathcal{S}}$ and $\operatorname{supp}\left(\overline{\mathbf{w}}\right)^{\prime}\subset\overline{\mathcal{S}}^{\prime}$, and such that

[TABLE]

are small enough, we can guarantee that $\overline{\mathbf{w}}$ and $\overline{\mathbf{w}}^{\prime}$ are close to each other.

In order to do so, we first establish a few simple properties and define relevant notations. Notice first that, we can apply the convolutional linear network to any input $u\in\mathbb{R}^{N|\mathcal{F}|}$, where $u$ is the (vertical) concatenation of the signals $u^{f}\in\mathbb{R}^{N}$ for $f\in\mathcal{F}$. Therefore, $M_{H}(\mathbf{w}_{H})\cdots M_{1}(\mathbf{w}_{1})$ is the (horizontal) concatenation of $|\mathcal{F}|$ matrices $Z^{f}\in\mathbb{R}^{N\times N}$ such that

[TABLE]

Let us consider the convolutional linear network defined by $\mathbf{w}\in\mathbb{R}^{S\times H}$ as well as $f\in\mathcal{F}$ and $n=1..N$. The column of $M_{H}(\mathbf{w}_{H})\cdots M_{1}(\mathbf{w}_{1})$ corresponding to the leaf $f$ and the entry $n$ is the translation by $n$ of

[TABLE]

where ${\mathcal{P}}(f)$ contains all the paths of ${\mathcal{P}}$ starting from the leaf $f$ and

[TABLE]

and we remind that $\mathcal{T}_{e^{h}}(\mathbf{w}_{h})$ is the convolution kernel on the edge $e^{h}$.

We define for any $h=1..H$ the mapping $\mathbf{e}_{h}:[\![S]\!]\longrightarrow\mathcal{E}(h)$ which provides for any $i=1..S$ the unique edge of $\mathcal{E}(h)$ such that the $i^{\mbox{th}}$ entry of $w\in\mathbb{R}^{S}$ contributes to $\mathcal{T}_{\mathbf{e}_{h}(i)}(w)$. Also, for any $\mathbf{i}\in[\![S]\!]^{H}$, we denote $\mathbf{p}_{\mathbf{i}}=(\mathbf{e}_{1}(\mathbf{i}_{1}),\ldots,\mathbf{e}_{H}(\mathbf{i}_{H}))$ and, for any $\mathcal{S}\in{{\mathcal{M}}}$,

[TABLE]

The latter contains all the indices of $\mathcal{S}$ corresponding to a valid path in the network. For any set of parameters $\mathbf{w}\in\mathbb{R}^{S\times H}$ and any path $\mathbf{p}\in{\mathcal{P}}$, we also denote by $\mathbf{w}^{\mathbf{p}}$ the restriction of $\mathbf{w}$ to its indices contributing to the kernels on the path $\mathbf{p}$. We also define, for any $\mathbf{i}\in[\![S]\!]^{H}$, $\mathbf{w}^{\mathbf{i}}\in\mathbb{R}^{S\times H}$ by

[TABLE]

so-that $P(\mathbf{w}^{\mathbf{i}})$ is a Dirac at position $\mathbf{i}$. The difference between $\mathbf{w}^{\mathbf{p}}$ and $\mathbf{w}^{\mathbf{i}}$ will not be ambiguous, once in context.

We can deduce from (17) that, when $\mathbf{i}\in\mathbf{I}_{\mathcal{S}}$, $M_{H}(\mathbf{w}^{\mathbf{i}}_{H})\cdots M_{1}(\mathbf{w}^{\mathbf{i}}_{1})$ simply convolves the entries at one leaf with a Dirac delta function. Therefore, all the entries of $M_{H}(\mathbf{w}^{\mathbf{i}}_{H})\cdots M_{1}(\mathbf{w}^{\mathbf{i}}_{1})$ are in $\{0,1\}$ and we denote \mathcal{D}_{\mathbf{i}}=\{(i,j)\in[\![N]\!]\times[\![N|\mathcal{F}|]\!]|\big{(}M_{H}(\mathbf{w}^{\mathbf{i}}_{H})\cdots M_{1}(\mathbf{w}^{\mathbf{i}}_{1})\big{)}_{i,j}=1\}.

We also denote $\mathbbm{1}\in\mathbb{R}^{S}$ a vector of size $S$ with all its entries equal to $1$. For any edge $e\in\mathcal{E}$, $\mathbbm{1}^{e}\in\mathbb{R}^{S}$ consists of zeroes except for the entries contributing to the convolution kernel on the edge $e$ which are equal to $1$. For any $\mathcal{S}\subset[\![S]\!]^{H}$, we define $\mathbbm{1}^{\mathcal{S}}\in\mathbb{R}^{S\times H}$ which consists of zeroes except for the entries corresponding to the indexes in $\mathcal{S}$ which are equal to $1$.

The equivalence relationship $\sim$, defined in Section 2, does not suffice to group parameters leading to the same network prediction. Indeed, with the considered convolutional networks, we can rescale the kernels on different path differently. Therefore, we say that two networks sharing the same architecture and defined by the parameters $\mathbf{w}$ and $\mathbf{w}^{\prime}\in\mathbb{R}^{S\times H}$ are equivalent if and only if

[TABLE]

The equivalence class of $\mathbf{w}\in\mathbb{R}^{S\times H}$ is denoted by $\{\mathbf{w}\}$. It is not difficult to see that the prediction of the networks defined by equivalent parameters are identical. For any $p\in[1,+\infty[$, we define

[TABLE]

where we remind that $d_{p}$ is defined in (3). Since $d_{p}$ is a metric, $\Delta_{p}$ is a metric between network classes.

The equivalence classes we have defined do not take into the account the fact it is possible to modify $\mathbf{w}$ in a way that corresponds to permutation of the nodes of the network. Taking into account this invariant is difficult and remains an open question. It has not been addressed in [Arora et al.(2014)Arora, Bhaskara, Ge, and Ma, Brutzkus and Globerson(2017), Li and Yuan(2017), Sedghi and Anandkumar(2014), Zhong et al.(2017)Zhong, Song, Jain, Bartlett, and Dhillon, Malgouyres and Landsberg(2016), Malgouyres and Landsberg(2019)].

Finally, we remind that because of (7), there exists a unique mapping

[TABLE]

such that

[TABLE]

where $P$ is the Segre embedding defined in (4).

Proposition 5**.**

Necessary condition of identifiability of a sparse network**

Only one of the two following alternatives can occur.

Either, there exist $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{{\mathcal{M}}}$ such that some entries of $M_{H}(\mathbbm{1}_{H}^{\mathcal{S}\cup\mathcal{S}^{\prime}})\cdots M_{1}(\mathbbm{1}_{1}^{\mathcal{S}\cup\mathcal{S}^{\prime}})M_{0}$ do not belong to $\{0,1\}$.

When this holds, $\{\overline{\mathbf{w}}\}$ is not always identifiable: there exists $\{\overline{\mathbf{w}}\}\neq\{\overline{\mathbf{w}}^{\prime}\}$ such that

[TABLE] 2. 2.

Or, for any $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{{\mathcal{M}}}$, all the entries of $M_{H}(\mathbbm{1}_{H}^{\mathcal{S}\cup\mathcal{S}^{\prime}})\cdots M_{1}(\mathbbm{1}_{1}^{\mathcal{S}\cup\mathcal{S}^{\prime}})M_{0}$ belong to $\{0,1\}$. When this holds :

1. (a)

For any $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{{\mathcal{M}}}$, for any distinct $\mathbf{i}\in\mathcal{S}$ and $\mathbf{i}^{\prime}\in\mathcal{S}^{\prime}$, we have $\mathcal{D}_{\mathbf{i}}\cap\mathcal{D}_{\mathbf{i}^{\prime}}=\emptyset$. 2. (b)

For any $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{{\mathcal{M}}}$, for any $\mathbf{w}\in\mathbb{R}^{S\times H}_{\mathcal{S}}$ and $\mathbf{w}^{\prime}\in\mathbb{R}^{S\times H}_{\mathcal{S}^{\prime}}$ and any distinct $\mathbf{p}$ and $\mathbf{p}^{\prime}\in{\mathcal{P}}$, we have

[TABLE] 3. (c)

If moreover $|{\mathcal{P}}|=1$ and $X$ is full row rank:

[TABLE]

The proof is in Appendix A.

Proposition 5, Item 1, expresses a necessary condition of stability: for any $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{{\mathcal{M}}}$, all the entries of $M_{H}(\mathbbm{1}_{H}^{\mathcal{S}\cup\mathcal{S}^{\prime}})\cdots M_{1}(\mathbbm{1}_{1}^{\mathcal{S}\cup\mathcal{S}^{\prime}})M_{0}$ belong to $\{0,1\}$. The condition is restrictive but not empty. We will see in the sequel that, when $X$ is full row rank, the condition is sufficient to guarantee the stability. Notice that the condition can be computed at a low cost by applying the network to Dirac delta functions, when $|{{\mathcal{M}}}|$ is not too large.

Proposition 6**.**

If $|{\mathcal{P}}|=1$ and $X$ is full row rank. If, for any $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{\mathcal{M}}$, all the entries of $M_{H}(\mathbbm{1}_{H}^{\mathcal{S}\cup\mathcal{S}^{\prime}})\cdots M_{1}(\mathbbm{1}_{1}^{\mathcal{S}\cup\mathcal{S}^{\prime}})M_{0}$ belong to $\{0,1\}$, then $\operatorname{Ker}\left(\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\right)$ is the orthogonal complement of $\mathbb{R}^{S^{H}}_{\mathcal{S}\cup\mathcal{S}^{\prime}}$ and $\mathcal{A}$ satisfies the sparse-deep-NSP with constants $(\gamma,\rho)=(1,+\infty)$ for ${\mathcal{M}}$. Moreover, $\sigma_{{\mathcal{M}}}=\sqrt{N}\sigma_{min}(X)$, where $\sigma_{min}(X)$ is the smallest singular value of $X$, is a deep-lower-RIP constant of $\mathcal{A}$ with regard to ${\mathcal{M}}$.

The proof of the proposition is in Appendix B.

Let us remind: we consider $X$ and $Y\in\mathbb{R}^{N\times n}$, $\overline{\mathcal{S}}$ and $\overline{\mathcal{S}}^{\prime}\in{{\mathcal{M}}}$ and parameters $\overline{\mathbf{w}}$ and $\overline{\mathbf{w}}^{\prime}\in\mathbb{R}^{S\times H}$ satisfying

[TABLE]

and denote

[TABLE]

where we will assume in the theorem that $\delta$ and $\eta$ are small.

For any path $\mathbf{p}\in{\mathcal{P}}$, we denote

[TABLE]

where we remind that $\overline{\mathbf{w}}^{\mathbf{p}}$ (resp ${\overline{\mathbf{w}}^{\prime}}^{\mathbf{p}}$) denotes the restriction of $\overline{\mathbf{w}}$ (resp $\overline{\mathbf{w}}^{\prime}$) to the path $\mathbf{p}$. Under the hypothesis of the following theorem, when $\delta+\eta$ is small, we will prove that $\delta^{\mathbf{p}}$ is small too for every $\mathbf{p}\in{\mathcal{P}}$ (see (25)).

Theorem 7**.**

Sufficient condition of stability**

Let $X$, $Y$, $\overline{\mathcal{S}}$, $\overline{\mathcal{S}}^{\prime}$, $\overline{\mathbf{w}}$, $\overline{\mathbf{w}}^{\prime}$, $\delta$ and $\eta$ be as described above (see (20) and (21)). Assume $X$ is full row rank.

If for any $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{\mathcal{M}}$, all the entries of $M_{H}(\mathbbm{1}_{H}^{\mathcal{S}\cup\mathcal{S}^{\prime}})\cdots M_{1}(\mathbbm{1}_{1}^{\mathcal{S}\cup\mathcal{S}^{\prime}})M_{0}$ belong to $\{0,1\}$ and if there exists $\varepsilon>0$ such that for all $e\in\mathcal{E}$, $\|\mathcal{T}_{e}(\overline{\mathbf{w}})\|_{\infty}\geq\varepsilon$ and for all $\mathbf{p}\in{\mathcal{P}}$, $\frac{\delta^{\mathbf{p}}}{\sqrt{N}\sigma_{min}(X)}\leq\frac{1}{2}\max(\|P(\overline{\mathbf{w}}^{\mathbf{p}})\|_{\infty},\|P((\overline{\mathbf{w}}^{\prime})^{\mathbf{p}})\|_{\infty})$, then $\overline{\mathbf{w}}$ and $\overline{\mathbf{w}}^{\prime}$ are close to each other: for any $p\in[1,\infty[$

[TABLE]

We remind that, according to Proposition 5, Item 1, the network is not identifiable when some entries of $M_{H}(\mathbbm{1}_{H}^{\mathcal{S}\cup\mathcal{S}^{\prime}})\cdots M_{1}(\mathbbm{1}_{1}^{\mathcal{S}\cup\mathcal{S}^{\prime}})M_{0}$ do not belong to $\{0,1\}$.

The proof of the theorem is in Appendix C.

5 Conclusion

We provide a necessary and sufficient condition of stability for the optimal weights of a sparse linear network. In the general setting, when no assumption is made on the architecture of the network, the stability constant $C$ is improved when compared to un-specified weight models [Malgouyres and Landsberg(2019)]. The gain is comparable to the gain obtained in compressed sensing when replacing the smallest singular value by the lower RIP constant [Elad(2010)]. We then specialize the results to sparse convolutional linear networks. In this analyses, we detail the stability condition in terms of a condition on the architecture and a condition on the sample inputs. The condition on the architecture is restrictive but not empty. The condition on the sample inputs is rather weak and basically requires to have as many (diverse) samples as the dimension of the input space. The constant $\sigma_{min}(X)$ is a key component of the stability constant.

\acks

Francois Malgouyres is funded by the project DEEL (\urlhttps://www.deel.ai/) and by the ANITI (\urlhttps://aniti.univ-toulouse.fr/index.php/en/). The author would like to thank Joseph Landsberg for all his remarks.

Appendix A Proof of Proposition 5

First notice that the entries of $M_{H}(\mathbbm{1}_{H}^{\mathcal{S}\cup\mathcal{S}^{\prime}})\cdots M_{1}(\mathbbm{1}_{1}^{\mathcal{S}\cup\mathcal{S}^{\prime}})$ are non-negative integers.

*Let us first assume that: * There exist $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{{\mathcal{M}}}$ and an entry of

[TABLE]

that does not belong to $\{0,1\}$.

Using (15), (16) and (17), we know that there is $n=1..N$ such that

[TABLE]

As a consequence, there is $\mathbf{i}$ and $\mathbf{j}\in{\mathcal{S}\cup\mathcal{S}^{\prime}}$ with $\mathbf{i}\neq\mathbf{j}$ and

[TABLE]

Therefore, since both ${\mathcal{T}}^{\mathbf{p}_{\mathbf{i}}}(\mathbf{w}^{\mathbf{i}})\,$ and ${\mathcal{T}}^{\mathbf{p}_{\mathbf{j}}}(\mathbf{w}^{\mathbf{j}})\,$ are Diracs,

[TABLE]

Since $\mathbf{i}\neq\mathbf{j}$, $\{\mathbf{w}^{\mathbf{i}}\}\neq\{\mathbf{w}^{\mathbf{j}}\}$ and the network is not identifiable. This proves Item 1.

*Let us now assume that: * For any $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{\mathcal{M}}$, all the entries of

[TABLE]

belong to $\{0,1\}$.

For any $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{\mathcal{M}}$ and any distinct $\mathbf{i}\in\mathcal{S}$ and $\mathbf{i}^{\prime}\in\mathcal{S}^{\prime}$, since ${\mathcal{T}}^{\mathbf{p}}(\mathbf{w}^{\mathbf{i}})\,$ and ${\mathcal{T}}^{\mathbf{p}}(\mathbf{w}^{\mathbf{i}^{\prime}})\,$ are Diracs, using (17), (16) and the hypothesis we establish Item 2a.

To prove Item 2b, we consider $\mathcal{S}$ and $\mathcal{S}^{\prime}\in{{\mathcal{M}}}$, $\mathbf{w}\in\mathbb{R}^{S\times H}_{\mathcal{S}}$ and $\mathbf{w}^{\prime}\in\mathbb{R}^{S\times H}_{\mathcal{S}^{\prime}}$, and distinct $\mathbf{p}\neq\mathbf{p}^{\prime}\in{\mathcal{P}}$. We have

[TABLE]

and

[TABLE]

Using the hypothesis, we know (as in the proof of Item 2a) that

[TABLE]

and conclude that Item 2b holds.

To prove the Item 2c, notice first that $(P(\mathbf{w}^{\mathbf{i}}))_{\mathbf{i}\not\in\mathbf{I}_{\mathcal{S}\cup\mathcal{S}^{\prime}}}$ forms a basis of $\{T\in\mathbb{R}^{S^{H}}|\forall\mathbf{i}\in\mathbf{I}_{\mathcal{S}\cup\mathcal{S}^{\prime}},T_{\mathbf{i}}=0\}$. We check using (17) and (9) that, for any $\mathbf{i}\not\in\mathbf{I}_{\mathcal{S}\cup\mathcal{S}^{\prime}}$,

[TABLE]

As a consequence,

[TABLE]

To prove the converse inclusion, we observe that

[TABLE]

where the last equality holds because, when $|{\mathcal{P}}|=1$, $X^{|\mathcal{F}|}=X$ is full row rank. Moreover, under the hypothesis of the proposition, for any distinct $\mathbf{i}$ and $\mathbf{j}\in\mathbf{I}_{\mathcal{S}\cup\mathcal{S}^{\prime}}$, $\mathcal{D}_{\mathbf{i}}\cap\mathcal{D}_{\mathbf{j}}=\emptyset$, and therefore

[TABLE]

Therefore, $\operatorname{rk}\left(\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\right)=|\mathbf{I}_{\mathcal{S}\cup\mathcal{S}^{\prime}}|$; i.e.

[TABLE]

and $\dim(\operatorname{Ker}\left(\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\right))=\dim(\{T\in\mathbb{R}^{S^{H}}|\forall\mathbf{i}\in\mathbf{I}_{\mathcal{S}\cup\mathcal{S}^{\prime}},T_{\mathbf{i}}=0\})$. Combined with (22), we obtain

[TABLE]

This proves Item 2c.

Appendix B Proof of Proposition 6

The fact that, $\operatorname{Ker}\left(\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\right)$ is the orthogonal complement of $\mathbb{R}^{S^{H}}_{\mathcal{S}\cup\mathcal{S}^{\prime}}$ is a direct consequence of Proposition 5, Item 2c, and the fact that, when $|{\mathcal{P}}|=1$, $\mathbf{I}_{\mathcal{S}\cup\mathcal{S}^{\prime}}=\mathcal{S}\cup\mathcal{S}^{\prime}$. We then deduce that, for any $T^{\prime}\in\operatorname{Ker}\left(\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\right)$, ${\mathbf{P}}_{\mathcal{S}\cup\mathcal{S}^{\prime}}T^{\prime}=0$. A straightforward consequence (see (11)) is that $\mathcal{A}$ satisfies the sparse-deep-NSP with constants $(\gamma,\rho)=(1,+\infty)$ for ${\mathcal{M}}$.

To calculate $\sigma_{{\mathcal{M}}}$, let us consider $\mathcal{S}$, $\mathcal{S}^{\prime}\in{\mathcal{M}}$ and $T$ in the orthogonal complement of $\operatorname{Ker}\left(\mathcal{A}_{\mathcal{S}\cup\mathcal{S}^{\prime}}\right)$. Using Proposition 5, Item 2c, we express $T$ under the form $T=\sum_{\mathbf{i}\in\mathcal{S}\cup\mathcal{S}^{\prime}}T_{\mathbf{i}}P(\mathbf{w}^{\mathbf{i}})$, where $\mathbf{w}^{\mathbf{i}}$ is defined by (18). Using (9) and (8), the linearity of $\mathcal{A}$ and the fact that, when $|{\mathcal{P}}|=1$, $X^{|\mathcal{F}|}=X$, we obtain

[TABLE]

Let us remind that, applying Proposition 5, Item 2a, the supports of $M_{H}(\mathbf{w}^{\mathbf{i}})\cdots M_{1}(\mathbf{w}^{\mathbf{i}})$ (i.e. $\mathcal{D}_{\mathbf{i}}$) and $M_{H}(\mathbf{w}^{\mathbf{j}})\cdots M_{1}(\mathbf{w}^{\mathbf{j}})$ (i.e. $\mathcal{D}_{\mathbf{j}}$) are disjoint, when $\mathbf{i}\neq\mathbf{j}$. Let us also add that, since $\mathcal{A}P(\mathbf{w}^{\mathbf{i}})$ is the matrix of a convolution with a Dirac mass, we have $|\mathcal{D}_{\mathbf{i}}|=N$, for all $\mathbf{i}\in\mathbf{I}$. Combining these two properties with (23) and reminding that $\|.\|$ is the Frobinius norm, we obtain

[TABLE]

Using that $P_{\mathcal{S}\cup\mathcal{S}^{\prime}}T=T$, we deduce the value of $\sigma_{{\mathcal{M}}}$ in the proposition.

Appendix C Proof of Theorem 7

Let us consider a path $\mathbf{p}\in{\mathcal{P}}$, using (17), since all the entries of $M_{H}(\mathbbm{1}_{H}^{\mathcal{S}\cup\mathcal{S}^{\prime}})\cdots M_{1}(\mathbbm{1}_{1}^{\mathcal{S}\cup\mathcal{S}^{\prime}})M_{0}$ belong to $\{0,1\}$, the restriction of the network to $\mathbf{p}$ satisfy the same property. Therefore, we can apply Proposition 6 and Theorem 3 to the restriction of the convolutional linear network to $\mathbf{p}$, with

[TABLE]

and obtain, when $\frac{\delta^{\mathbf{p}}}{\sqrt{N}\sigma_{min}(X)}\leq\frac{1}{2}\max(\|P(\overline{\mathbf{w}}^{\mathbf{p}})\|_{\infty},\|P((\overline{\mathbf{w}}^{\prime})^{\mathbf{p}})\|_{\infty})$, for any $p\in[1,\infty[$

[TABLE]

We also have, using the definition of $X^{|\mathcal{F}|}$,

[TABLE]

where the penultimate equality is due to Proposition 5, Item 2b. Combining this inequality, the definition of $\delta^{\mathbf{p}}$ and a standard norm inequality, we obtain

[TABLE]

Finally, combining the definition of the metric $\Delta_{p}$ (19), (24) and the above inequality we obtain

[TABLE]