Designing recurrent neural networks by unfolding an l1-l1 minimization algorithm

TL;DR

This paper introduces a novel RNN architecture derived from unfolding an l1-l1 minimization algorithm, specifically designed for sequential signal reconstruction, leveraging sparsity in signals and their differences.

Contribution

The paper presents a new RNN model based on unfolding a proximal gradient method for l1-l1 minimization, tailored for sparse sequential signal reconstruction.

Findings

Outperforms state-of-the-art RNN models in video frame reconstruction from compressive measurements.

Demonstrates the effectiveness of unfolding optimization algorithms into neural network architectures.

Leverages sparsity in signals and their differences for improved reconstruction accuracy.

Abstract

We propose a new deep recurrent neural network (RNN) architecture for sequential signal reconstruction. Our network is designed by unfolding the iterations of the proximal gradient method that solves the l1-l1 minimization problem. As such, our network leverages by design that signals have a sparse representation and that the difference between consecutive signal representations is also sparse. We evaluate the proposed model in the task of reconstructing video frames from compressive measurements and show that it outperforms several state-of-the-art RNN models.

Tables

Table 1. Table 1 : Average PSNR results [in dB] for sequential frame reconstruction on the testing dataset.

Model	Compression rate
	50%	33%	25%	20%
Stacked-RNN	38.56	35.82	33.12	30.78
Stacked-LSTM	37.02	34.06	31.55	29.60
Stacked-GRU	40.3	37.31	33.98	31.09
SISTA-RNN	42.52	37.20	33.85	30.91
Our model	44.65	38.90	34.22	30.76

Equations

$$\underset{\mathbf{h}}{\min }\frac{1}{2}\Vert \mathbf{x}-\mathbf{AD}\mathbf{h}{\Vert }_2^2+\lambda \Vert \mathbf{h}{\Vert }_1,$$

$${\mathbf{h}}^{(i)}={\varphi }_{\frac{\lambda }{a}}({\mathbf{h}}^{(i-1)}-\frac{1}{a}{\mathbf{D}}^T{\mathbf{A}}^T(\mathbf{AD}{\mathbf{h}}^{(i-1)}-\mathbf{x})),$$

$${\boldsymbol{h}}_{t}^{(k)}=\left\{\begin{array}[]{l}\phi\Big{(}\mathbf{W}^{(1)}{\boldsymbol{h}}_{t-1}^{(1)}+\mathbf{U}{\boldsymbol{x}}_{t}\Big{)},~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}k=1,\\ \phi\Big{(}\mathbf{W}^{(k)}{\boldsymbol{h}}_{t-1}^{(k)}+\mathbf{S}^{(k)}{\boldsymbol{h}}_{t}^{(k-1)}\Big{)},~{}k>1,\\ \end{array}\right.$$

$$\widehat{{\mathbf{s}}_t}=\mathbf{V}{\mathbf{h}}_t^{(K)}+{\mathbf{b}}_t,$$

$$\underset{{\mathbf{h}}_t}{\min }\frac{1}{2}\Vert {\mathbf{x}}_t-\mathbf{AD}{\mathbf{h}}_t{\Vert }_2^2+{\lambda }_1\Vert {\mathbf{h}}_t{\Vert }_1+\frac{{\lambda }_2}{2}\Vert \mathbf{D}{\mathbf{h}}_t-\mathbf{FD}{\mathbf{h}}_{t-1}{\Vert }_2^2,$$

$$\underset{{\mathbf{h}}_t}{\min }\frac{1}{2}\Vert {\mathbf{x}}_t-\mathbf{AD}{\mathbf{h}}_t{\Vert }_2^2+{\lambda }_1\Vert {\mathbf{h}}_t{\Vert }_1+{\lambda }_2\Vert {\mathbf{h}}_t-\text{G}{\mathbf{h}}_{t-1}{\Vert }_1,$$

$${\varphi }_{{\gamma }_1,{\gamma }_2,v}(u)=\{\begin{array}{ll}u-{\gamma }_1-{\gamma }_2,& v+{\gamma }_1+{\gamma }_2\le u<\infty \\ v,& v+{\gamma }_1-{\gamma }_2\le u<v+{\gamma }_1+{\gamma }_2\\ u-{\gamma }_1+{\gamma }_2,& {\gamma }_1-{\gamma }_2\le u<v+{\gamma }_1-{\gamma }_2\\ 0,& -{\gamma }_1-{\gamma }_2\le u<{\gamma }_1-{\gamma }_2\\ u+{\gamma }_1+{\gamma }_2,& -\infty <u<-{\gamma }_1-{\gamma }_2\end{array}$$

$${\varphi }_{{\gamma }_1,{\gamma }_2,v}(u)=\{\begin{array}{ll}u-{\gamma }_1-{\gamma }_2,& {\gamma }_1+{\gamma }_2\le u<\infty \\ 0,& -{\gamma }_1+{\gamma }_2\le u<{\gamma }_1+{\gamma }_2\\ u+{\gamma }_1-{\gamma }_2,& v-{\gamma }_1+{\gamma }_2\le u<-{\gamma }_1+{\gamma }_2\\ v,& v-{\gamma }_1-{\gamma }_2\le u<v-{\gamma }_1+{\gamma }_2\\ u-{\gamma }_1+{\gamma }_2,& -\infty <u<v-{\gamma }_1-{\gamma }_2,\end{array}$$

$${\boldsymbol{h}}_{t}^{(k)}=\left\{\begin{array}[]{l}\phi_{\frac{\lambda_{1}}{\alpha},\frac{\lambda_{2}}{\alpha},\textbf{Gh}^{(K)}_{t-1}}\Big{(}\mathbf{W}^{(1)}{\boldsymbol{h}}_{t-1}^{(1)}+\mathbf{U}{\boldsymbol{x}}_{t}\Big{)},~{}\text{if}~{}k=1,\\ \phi_{\frac{\lambda_{1}}{\alpha},\frac{\lambda_{2}}{\alpha},\textbf{Gh}^{(K)}_{t-1}}\Big{(}\mathbf{S}^{(k)}{\boldsymbol{h}}_{t}^{(k-1)}+\textbf{U}{\boldsymbol{x}}_{t}\Big{)},~{}\text{if}~{}k>1,\\ \end{array}\right.$$

$$\widehat{{\mathbf{s}}_t}=\mathbf{V}{\mathbf{h}}_t^{(K)}+{\mathbf{b}}_t.$$

$$\mathbf{U}=\frac{1}{\alpha }{\mathbf{D}}^T{\mathbf{A}}^T,\forall k,$$

$${\mathbf{W}}^{(1)}=\text{G}-\frac{1}{\alpha }{\text{D}}^T{\text{A}}^T\text{AD}\text{G},$$

$${\mathbf{S}}^{(k)}=\mathbf{I}-\frac{1}{\alpha }{\mathbf{D}}^T{\mathbf{A}}^T\mathbf{AD},k>1,$$

$$\mathbf{V}=\mathbf{D},{\mathbf{b}}_t=0.$$

Full text

**Index Terms— ** Sparse signal recovery, deep unfolding, recurrent neural networks, $\ell_{1}\text{-}\ell_{1}$ minimization.

1 Introduction

The problem of reconstructing sequential signals from low-dimensional—and possibly corrupted—observations across time appears in various imaging applications, including compressive video sensing [1], dynamic magnetic resonance imaging [2], and mm-Wave imaging [3]. When reconstructing time-varying signals, one needs to leverage prior knowledge; namely that (i) at a given time instance the signal has a low-complexity representation, such as sparsity in a learned dictionary or fixed basis, and (ii) signals (or their representations) across time are correlated (temporal correlation).

Various methods for sequential signal reconstruction have been proposed in the past. The method in [4] adapted a Kalman filter in sequential compressed sensing, whereas the Modified-CS method [5] integrates an estimate the signal’ s support into the reconstruction scheme. Alternatively, the methods in [6, 7] considered that two consecutive sparse signal representations are close under an $\ell_{1}$- or $\ell_{2}$-norm metric. Such approaches, however, recover the signals using iterative optimization algorithms, leading to high computational complexity when the dimensionality of the problem increases.

Deep neural networks (DNNs) have recently achieved state-of-the-art performance in solving inverse problems [8]. These approaches come with the additional benefit of fast reconstruction as they do not have to solve an optimization problem during inference. However, DNNs are black-box models, meaning that they do not integrate prior or domain knowledge, and thus lack interpretability and theoretical guarantees [8]. Recent efforts on designing DNNs that incorporate domain knowledge, include deep unfolding methods, which interpret a DNN as an unrolled version of an iterative optimization algorithm. Examples include the learned-ISTA (LISTA) network [9], which unfolds the iterative soft-thresholding algorithm (ISTA) [10], the unfolded versions of the approximate message passing [11] and iterative hard thresholding [12] algorithms, and the ADMM-Net [13].

Little attention has, however, been devoted to the design of deep recurrent neural networks (RNNs) [14] for representing sequential signals. The authors of [15] proposed an RNN design—named SISTA-RNN—by unfolding the sequential version of ISTA. The problem that the algorithm solves considers that two consecutive signal realizations are close in the $\ell_{2}$-norm sense. In this work, we propose a novel RNN model for sequential signal recovery. Our model is derived by unfolding a proximal gradient method that solves the $\ell_{1}\text{-}\ell_{1}$ minimization problem [6, 7]. This problem assumes that the different between sequential sparse signal representations is also sparse, and it is proven to outperform both $\ell_{1}$ and $\ell_{1}\text{-}\ell_{2}$ minimization [16]. We apply the proposed model in the problem of video reconstruction from low-dimensional measurements, that is, sequential frame compressed sensing. Experimentation on the moving MNIST dataset111The source code to replicate our experiments is available on https://github.com/dhungle.[[17](#bib.bib17)] shows that the proposed model achieves higher reconstruction results compared to various state-of-the-art RNN models, including SISTA-RNN [15].

The paper is organized as follows. Section 2 presents the background of the work. Section 3 describes the proposed model for video reconstruction. Section 4 presents the experiments and section 5 concludes the work.

2 Background and Related Work

2.1 Sparse signal reconstruction

Consider the problem of reconstructing a sparse signal $\boldsymbol{s}\in\mathbb{R}^{n}$ from noisy measurements ${\boldsymbol{x}}=\mathbf{A}\boldsymbol{s}+\boldsymbol{\eta}$, where $\mathbf{A}\in\mathbb{R}^{m\times n}~{}(m\ll n)$ is a sensing matrix and $\boldsymbol{\eta}$ a noise vector. By leveraging that $\boldsymbol{s}$ has a sparse representation ${\boldsymbol{h}}\in\mathbb{R}^{d}$ in a dictionary $\mathbf{D}\in\mathbb{R}^{n\times d}$, that is, $\boldsymbol{s}=\mathbf{D}{\boldsymbol{h}}$, the signal can be recovered from the measurements by solving [18]:

[TABLE]

where $\|\cdot\|_{p}$ is the $\ell_{p}$-norm and $\lambda$ is a regularization parameter. ISTA [10] solves (1) by iterating over

[TABLE]

where $\phi_{\gamma}(u)=\mathrm{sign}(u)(0,|u|-\gamma)_{+}$ is the soft thresholding operator [see Fig.1], $\gamma=\frac{\lambda}{a}$, and $a$ is an upper bound on the Lipschitz constant of the gradient of $\frac{1}{2}\|{\boldsymbol{x}}-\mathbf{A}\mathbf{D}{\boldsymbol{h}}\|_{2}^{2}$. LISTA [9] unrolls the iterations of ISTA into a feedforward neural network with shared weights, where each layer implements an iteration: ${\boldsymbol{h}}^{(i)}=\phi_{\gamma}(\mathbf{\Xi}{\boldsymbol{h}}^{(i-1)}+\mathbf{Z}{\boldsymbol{x}}),$ with $\mathbf{\Xi}=\mathbf{I}-\frac{1}{a}\mathbf{D}^{T}\mathbf{A}^{T}\mathbf{A}\mathbf{D}$, $\mathbf{Z}=\frac{1}{a}\mathbf{D}^{T}\mathbf{A}^{T}$, and $\gamma$ learned from data.

2.2 Stacked-RNNs for sequential signal representation

RNNs are connectionist models with self-feedback loops allowing information to pass across sequential steps. In a stacked-RNN [14], see Fig. 2(a), the vertical stack of network layers capture the latent representation of an input signal at a given time instance and the horizontal connections learn the temporal relationship across signals. RNNs can be used to recover a sequence of signals $\boldsymbol{s}_{t}$, $t=1,2,\dots,$ from a sequence of noisy measurement vectors ${\boldsymbol{x}}_{t}=\mathbf{A}\boldsymbol{s}_{t}+\boldsymbol{\eta}_{t}$. Specifically, given ${\boldsymbol{x}}_{t}$, $t=1,2,\dots$, the signal representation at the $k^{th}$ layer of $K$ layers, ${\boldsymbol{h}}_{t}^{(k)}$, and the reconstructed signal, $\hat{\boldsymbol{s}_{t}}$ at time $t$ are calculated as

[TABLE]

[TABLE]

where $\phi(\cdot)$ is a nonlinear activation function such as the tanh or the ReLU, $\mathbf{W},\mathbf{S},~{}\mathbf{U},~{}\mathbf{V}$ are affine transformations, and $\boldsymbol{b}_{t}$ are output bias vectors. These parameters, along with initial hidden states ${\boldsymbol{h}}_{0}$, can be trained using gradient descent with backpropagation-through-time. Different RNN architectures can be applied to solve the problem, e.g., the long short-term memory (LSTM) network [19] and the gated recurrent unit (GRU) [20], which have been proposed to prevent the vanishing gradient problem with long input sequences.

2.3 The SISTA stacked-RNN network

Traditional RNN models [19, 20, 14] do not integrate the knowledge that the signals $\boldsymbol{s}_{t}$ have sparse representations ${\boldsymbol{h}}_{t}$, $t=1,2,\dots$. To address this issue, the study in [15] proposed a stacked-RNN architecture, which stems from unfolding an iterative soft-thresholding algorithm—referred to as SISTA—that solves the following problem:

[TABLE]

where $\mathbf{F}\in\mathbb{R}^{n\times n}$ is a correlation matrix between $\boldsymbol{s}_{t}$ and $\boldsymbol{s}_{t-1}$, and $\lambda_{1},\lambda_{2}$ are regularization parameters. Similar to LISTA [9], the SISTA-RNN network [15] uses the soft-thresholding operator $\phi_{\gamma}(u)$ [Section 2.1 and Fig.1] as activation function.

3 The proposed RNN via $\ell_{1}\text{-}\ell_{1}$ minimization

This section describes the proposed RNN model, which stems from unfolding the steps of a proximal method that solves the $\ell_{1}\text{-}\ell_{1}$ minimization problem.

3.1 $\ell_{1}$-$\ell_{1}$ minimization in sequential signal recovery

In sequential signal recovery, one can recover the signal $\boldsymbol{s}_{t}=\mathbf{D}{\boldsymbol{h}}_{t}$ from measurements ${\boldsymbol{x}}_{t}$ by solving the $\ell_{1}\text{-}\ell_{1}$ minimization problem [6, 7]:

[TABLE]

where $\textbf{G}\in\mathbb{R}^{d\times d}$ is an affine transformation that promotes the correlation between the sparse representations between two consecutive instantiations of the signal, ${\boldsymbol{h}}_{t-1}$ and ${\boldsymbol{h}}_{t}$, and $\lambda_{1},\lambda_{2}>0$ are regularization parameters. We highlight that in (6) the correlation between ${\boldsymbol{h}}_{t}$ and $\textbf{G}{\boldsymbol{h}}_{t-1}$ is encoded using the $\ell_{1}$-norm as opposed to using the $\ell_{2}$-norm to express the correlation between $\boldsymbol{s}_{t}$ and $\textbf{F}\boldsymbol{s}_{t-1}$ in SISTA [see (5)]. The motivation is twofold: Firstly, it is proven that $\ell_{1}$-$\ell_{1}$ outperforms $\ell_{1}$-$\ell_{2}$ minimization in sparse signal recovery [16]. Secondly, from an application perspective, we know that the error between consecutive video frames (or their motion-compensated versions) typically follow the Laplace rather than the Gaussian distribution [21].

The objective function of (6) consists of two parts: the differentiable function $f({\boldsymbol{h}}_{t})=\frac{1}{2}\|{\boldsymbol{x}}_{t}-\mathbf{A}\mathbf{D}{\boldsymbol{h}}_{t}\|_{2}^{2}$ and the non-smooth function $g({\boldsymbol{h}}_{t})=\lambda_{1}\|{\boldsymbol{h}}_{t}\|_{1}+\lambda_{2}\|{\boldsymbol{h}}_{t}-\textbf{G}{\boldsymbol{h}}_{t-1}\|_{1}$. Hence, in this work, we propose to solve (6) using a proximal gradient method, the steps of which are given in Algorithm 1. In our algorithm, step 5 applies a gradient descent update for $f({\boldsymbol{h}}_{t})$ with a learning rate $\frac{1}{a}>0$ and step 6 applies element-wise the proximal operator for our problem, $\phi_{\frac{\lambda_{1}}{\alpha},\frac{\lambda_{2}}{\alpha},\textbf{Gh}^{(K)}_{t-1}}({\boldsymbol{u}})$. For notation brevity, we denote $\gamma_{1}=\frac{\lambda_{1}}{\alpha}$, $\gamma_{2}=\frac{\lambda_{2}}{\alpha}$ and $u$, $v$ an element in ${\boldsymbol{u}}$, $\textbf{Gh}^{(K)}_{t-1}$, respectively; the proximal operator is then defined as:

[TABLE]

if $v\geq 0$, and

[TABLE]

if $v<0$. Fig. 1 depicts a schema of the proximal operator of our algorithm for $v\geq 0$ [see Fig. 1] in comparison with the soft-thresholding operator $\phi_{\gamma}(u)$ [see Fig. 1], which is used in SISTA [15].

3.2 The proposed $\ell_{1}$-$\ell_{1}$-RNN architecture

We now describe the proposed stacked-RNN architecture for sequential signal recovery, which we call $\ell_{1}$-$\ell_{1}$-RNN. The network, which is shown in Fig. 2(b), is designed by unrolling the steps of Algorithm 1 across the iterations $k=1,\dots,K$ (yielding the hidden layers) and time instances $t=1,\dots,T$. Specifically, the $k$-th hidden layer is given by

[TABLE]

and the reconstructed signal at time instance $t$ is calculated as

[TABLE]

where $\mathbf{U}$, $\mathbf{W}$, $\mathbf{S}^{(k)}$, $\mathbf{V}$ are defined as

[TABLE]

The activation function has the form of the proximal operator $\phi_{\gamma_{1},\gamma_{2},v}(u)$ with the parameters $\gamma_{1}$, $\gamma_{2}$ learned during training. We train our network in an end-to-end fashion: Vectorized frames are inputs $\boldsymbol{s}_{t}$, $t=1,\dots,T$, which are compressed by a linear measurement layer A, resulting in compressive measurements ${\boldsymbol{x}}_{t}$. The reconstructed frames $\hat{\boldsymbol{s}}_{t}$ are obtained by multiplying linearly the hidden representation ${\boldsymbol{h}}_{t}^{(K)}$ with the dictionary D. During training, we minimize the loss function $\mathcal{L}=\sum_{t=1}^{T}\|\boldsymbol{s}_{t}-\hat{\boldsymbol{s}_{t}}\|_{2}^{2}+\beta\|\mathbf{\theta}\|_{2}^{2}$ using Adam optimization [22] on mini-batches, where the trainable parameters are $\mathbf{\theta}=\{\textbf{A},\textbf{D},\textbf{G},{\boldsymbol{h}}_{0},\alpha,\lambda_{1},\lambda_{2}\}$ and $\beta$ is the hyper-parameter of the weight decay regularization.

4 Experiments

We assess the performance of the proposed RNN model in the problem of video frame reconstruction from compressive measurements. We use the moving MNIST dataset [17] for our experiments; the 10.000 video sequences (with 20 frames each) in the dataset are split into non-overlapping training, validation, and test sets, consisting of 8.000, 1.000, and 1.000 sequences, respectively. In order to reduce the computational complexity and memory requirements, each frame has been spatially downscaled from $64\times 64$ to $16\times 16$ pixels using bilinear decimation. After vectorising, we obtain signal sequences of $\boldsymbol{s}_{1:T}\in\mathbb{R}^{256}$ with $T=20$ time steps. For each sequence, we obtain a sequence of measurements ${\boldsymbol{x}}_{1:20}$ using a trainable linear sensing matrix $\textbf{A}\in\mathbb{R}^{m\times n}$, with $n=256$ and $m<n$. We test several different values of $m$ corresponding to compression rates of $\{50\%,33\%,25\%,20\%\}$. The dictionary $\textbf{D}\in\mathbb{R}^{256\times 1024}$ is initialized with the overcomplete discrete cosine transform (DCT).

We compare the reconstruction performance of the proposed RNN model against existing various RNN models, namely, SISTA-RNN [15], stacked-RNN [23], stacked-LSTM (with the LSTM cell architecture from [19]), and stacked-GRU (with the GRU cell architecture from [20]). Following the experimental setup in [15], we initialize the sparse code ${\boldsymbol{h}}_{0}^{(K)}$ as the zero vector and $\alpha=1.0$. We empirically found that using a weight decay $\beta=0.01$ in the SISTA-RNN and the proposed $\ell_{1}$-$\ell_{1}$-RNN yields best results in the validation set, whereas for the rest of the models not using a weight decay ($\beta=0$) provides best results. In order to initialize $\lambda_{1},\lambda_{2}$ [see (6)], we perform a random search to find the best combination that yields high reconstruction accuracy in the validation dataset, and obtain $\lambda_{1}=1.0$, $\lambda_{2}=0.01$ for our model and $\lambda_{1}=0.3$, $\lambda_{2}=0.01$ for SISTA-RNN. We use $K=3$ stacks for all models, except for the stacked-LSTM model222The stacked-LSTM with $K=3$ could not be trained from the data, possibly due to the large amount of parameters to optimise. for which we set $K=2$. All weights and biases are initialized with a uniform distribution of $\pm\frac{1}{\sqrt{d}}$, where $d=1024$ is the size of each hidden layer. We train the networks for 200 epochs with a learning rate of 0.0003, and a batch size of 32.

Table 1 reports the reconstructed PSNR results averaged across all frames and sequences in the test set. The experiments show that our model outperforms all other models at compression rates $50\%,33\%$, and $25\%$, bringing respective improvements of 2.13 dB, 1.59 dB, 0.24 dB over the second best model. At the rate of $20\%$, the proposed model is outperformed by SISTA-RNN and stacked-GRU. In addition, Fig. 3 illustrates average PSNR (measured on the validation set) versus training epochs curves for all models at a compression rate of $33\%$. The learning curve of the proposed RNN is consistently better than those of the other models.

After 200 epochs of training our model, we measured the sparsity (number of zero elements) of 55% in the last layer ${\boldsymbol{h}}_{t}^{(3)}$. We leave the investigation of how sparsity affects reconstruction performance for future work.

5 Conclusion

We proposed a stacked RNN for sequential sparse signal recovery from compressive measurements. Our RNN architecture incorporates prior knowledge about the structure of the signals and their correlation by deep-unfolding a proximal gradient method for the $\ell_{1}\text{-}\ell_{1}$ minimization problem. Our experiments in the task of video-frame recovery from compressive measurement show that our model outperforms several state-of-the-art RNNs.