An Underparametrized Deep Decoder Architecture for Graph Signals

TL;DR

This paper introduces a novel underparametrized deep decoder architecture tailored for graph signals, leveraging graph coarsening for improved reconstruction on irregular domains, outperforming traditional methods.

Contribution

It generalizes untrained deep decoders to graph signals by integrating graph-aware upsampling and hierarchical clustering, enhancing reconstruction performance.

Findings

Graph-aware upsampling improves reconstruction accuracy.

Incorporating graph topology significantly outperforms methods ignoring structure.

The approach is effective on both synthetic and real-world datasets.

Abstract

While deep convolutional architectures have achieved remarkable results in a gamut of supervised applications dealing with images and speech, recent works show that deep untrained non-convolutional architectures can also outperform state-of-the-art methods in several tasks such as image compression and denoising. Motivated by the fact that many contemporary datasets have an irregular structure different from a 1D/2D grid, this paper generalizes untrained and underparametrized non-convolutional architectures to signals defined over irregular domains represented by graphs. The proposed architecture consists of a succession of layers, each of them implementing an upsampling operator, a linear feature combination, and a scalar nonlinearity. A novel element is the incorporation of upsampling operators accounting for the structure of the supporting graph, which is achieved by considering a systematic graph coarsening approach based on hierarchical clustering. The numerical results carried out in synthetic and real-world datasets showcase that the reconstruction performance can improve drastically if the information of the supporting graph topology is taken into account.

Equations

$$\mathbf{x}={\mathbf{V}}_K{\tilde{\mathbf{x}}}_K,$$

$$\mathbf{x}=f_{\mathbf{\Theta }}(\mathbf{Z}),$$

$${\mathbf{Y}}^{(0)}$$

$${\hat{\mathbf{Y}}}^{(l)}$$

$${\check{\mathbf{Y}}}_{ij}^{(l)}$$

$${\mathbf{Y}}^{(l)}$$

$$\mathbf{x}$$

$${\mathcal{T}}_{{\mathbf{\varphi }}^{(l)}}^{(l)}\{{\mathbf{Y}}^{(l-1)}\}={\mathbf{U}}^{(l)}{\mathbf{Y}}^{(l-1)}{{\mathbf{\Phi }}^{(l)}}^T.$$

$${\mathbf{U}}^{(l)}=(\gamma \mathbf{I}+(1-\gamma ){\mathbf{A}}^{(l)}){\mathbf{P}}^{(l)},$$

Full text

**Index Terms— ** Deep learning, deep decoders, graph signal processing, graph upsampling, hierarchical clustering.

1 Introduction

Ours is a digital and pervasively connected world where an unprecedented amount of information is generated, recorded and processed. Coping with the challenges associated with complex contemporary datasets calls for new models and tools able to, among other things, account for irregular information domains and nonlinear interactions. Motivated by these goals, this paper leverages recent successes in the fields of graph signal processing (GSP) and deep learning (DL) to propose a new architecture for (underparametrized) deep decoding of signals supported on graphs.

The main goal of GSP is to generalize classical SP results to signals defined on irregular domains [1, 2]. The underlying assumption in GSP is twofold: i) the geometry of the irregular domain can be well represented by a graph (i.e., by local pairwise interactions) and ii) the properties of the signal of interest depend on the topology of such a graph. A plethora of graph signals exist, with examples ranging from neurological activity patterns defined on top of brain networks to the spread of epidemics over social networks [1]. Assuming complete knowledge of the graph, early GSP efforts focused on the rigorous definition and analysis of linear graph filters, graph Fourier representations, and their application to inverse problems such as denoising, deconvolution, or sampling and reconstruction, to name a few [3, 4, 5, 6]. More recent efforts focus on identifying the supporting graph, developing robust GSP schemes, analyzing higher-dimensional graph signals, and developing nonlinear GSP architectures. Tractable and useful examples of the latter category are median filters [7] and DL schemes [8], which have been mostly focused on (graph-filter based) convolutional architectures [9, 10], see, e.g., [11, 12] for some exceptions. The interest in DL architectures is not surprising since in the last years, propelled by advances in data availability, computation, and optimization, deep neural network architectures have emerged as one of the most powerful tools for machine learning over modern datasets [13]. Early works addressed classification and regression tasks, especially for speech and images, where convolutional architectures have shown a remarkable performance [13, 14]. Subsequent works went beyond supervised tasks, including representation learning, reinforcement learning and development of deep latent models. Two notable examples within this later category are generative (adversarial) networks and deep autoencoders [15, 16, 17], which are extremely useful to address inverse problems where a low-dimensional representation of the signal of interest is required [18]. Generally, most of the success of DL for image and speech processing is attributed to the high availability of data, which enables the training of overparametrized architectures. However, it was recently shown that the structure of convolutional [19] and non-convolutional [20] decoder (generator) networks can effectively capture many low-level image features even prior to any training.

Inspired by these lines of work, our main contribution is the design of the first underparametrized deep decoder architecture for graph signals. By leveraging agglomerative hierarchical clustering methods, the proposed architectures provide a topology-aware nonlinear representation basis for graph signals that is more efficient than the graph Fourier transform (GFT) and than can be used for tasks such as compression, denoising, and inpainting of signals on graphs.

2 Preliminaries

This section introduces notation and reviews basic GSP and deep decoding concepts that will be used in the ensuing sections.

2.1 Graph Signal Processing

Let ${\mathcal{G}}=({\mathcal{N}},{\mathcal{E}})$ denote a directed graph, where ${\mathcal{N}}$ is the set of nodes, with cardinality $N$, and ${\mathcal{E}}$ is the set of edges, with $(i,j)\in{\mathcal{E}}$ if $i$ is connected to node $j$. The set ${\mathcal{N}}_{i}:=\{\,j\,|(j,i)\in{\mathcal{E}}\}$ denotes the incoming neighborhood of node $i$. For a given ${\mathcal{G}}$, the adjacency matrix ${\mathbf{A}}\in{\mathbb{R}}^{N\times N}$ is sparse with non-zero elements $A_{ij}$ if and only if $(j,i)\in{\mathcal{E}}$. If ${\mathcal{G}}$ is unweighted, the elements $A_{ij}$ are binary. If the graph is weighted, then the value of $A_{ij}$ captures the strength of the link from $j$ to $i$. The focus of this paper is not on analyzing ${\mathcal{G}}$, but a graph signal defined on its set of nodes. Such a signal can be represented as a vector ${\mathbf{x}}=[x_{1},\ldots,x_{N}]^{T}\in\mathbb{R}^{N}$ where the $i$-th entry represents the signal value at node $i$. Since the signal ${\mathbf{x}}$ is defined on the graph ${\mathcal{G}}$, the underlying assumption in GSP is that the properties of ${\mathbf{x}}$ depend on the topology of ${\mathcal{G}}$. For instance, if the graph encodes similarity and the value of $A_{ij}$ is high, then one would expect the signal values $x_{i}$ and $x_{j}$ to be closely related.

The graph-shift operator (GSO). The GSO ${\mathbf{S}}$ is defined as an $N\times N$ matrix whose entry $S_{ij}$ can be non-zero only if $i=j$ or $(j,i)\in{\mathcal{E}}$. Common choices for ${\mathbf{S}}$ are ${\mathbf{A}}$ and the graph Laplacian ${\mathbf{L}}$, which is defined as ${\mathbf{L}}:=\text{diag}({\mathbf{A}}{\mathbf{1}})-{\mathbf{A}}$ [1, 2]. The GSO accounts for the topology of the graph and, at the same time, represents a linear transformation that can be computed locally. Specifically, if ${\mathbf{y}}=[y_{1},\ldots,y_{N}]^{T}$ is defined as ${\mathbf{y}}={\mathbf{S}}{\mathbf{x}}$, then node $i$ can compute $y_{i}$ provided that it has access to the values of $x_{j}$ at its neighbors $j\in{\mathcal{N}}_{i}$. We assume that ${\mathbf{S}}$ is diagonalizable so that there exists an $N\times N$ matrix ${\mathbf{V}}$ and a diagonal matrix $\boldsymbol{\Lambda}$ such that ${\mathbf{S}}={\mathbf{V}}\boldsymbol{\Lambda}{\mathbf{V}}^{-1}$.

Frequency domain representation. Key to define the frequency representation of graph signals and graph filters is the eigendecomposition of ${\mathbf{S}}$. In particular, the frequency representation of a graph signal ${\mathbf{x}}$ is defined as $\tilde{{\mathbf{x}}}:={\mathbf{V}}^{-1}{\mathbf{x}}$ , with ${\mathbf{V}}^{-1}$ acting as the GFT [3]. Leveraging the GFT, one can the generalize the notion of bandlimitedness to the graph domain. Specifically, it is said that ${\mathbf{x}}$ is a low-pass bandlimited graph signal if it holds that its frequency representation ${\tilde{\mathbf{x}}}={\mathbf{V}}^{-1}{\mathbf{x}}$ satisfies that ${\tilde{x}}_{k}=0$ for all $k>K$, where $K\leq N$ represents the bandwidth of the signal ${\mathbf{x}}$. Note that if ${\mathbf{x}}$ is bandlimited with bandwidth $K$, then it holds that

[TABLE]

where ${\tilde{\mathbf{x}}}_{K}:=[{\tilde{x}}_{1},...,{\tilde{x}}_{K}]$ collects the active frequency components and the $N\times K$ matrix ${\mathbf{V}}_{K}:=[{\mathbf{v}}_{1},...,{\mathbf{v}}_{K}]$ collects the corresponding eigenvectors. In other words, (1) states that: i) the original $N$-dimensional signal ${\mathbf{x}}$ lives on a (reduced-dimensionality) subspace related to the spectrum of the supporting graph; and ii) the $K$ values in ${\tilde{\mathbf{x}}}_{K}$ suffice to represent ${\mathbf{x}}$. This reduced-dimensionality representation has been exploited to design efficient algorithms for denoising, reconstruction from a subset of samples, and other inverse problems dealing with graph signals [21, 4]. While the simple linear model in (1) has been shown relevant in real-world applications [2], many modern datasets have a nonlinear latent structure that linear reduced-dimensionality models such as the one in (1) are not able to capture.

2.2 Deep Decoding

Deep decoders are capable of learning nonlinear reduced dimensionality representations to approximate a dataset (collection of signals) of interest. Succinctly, the idea behind deep decoding is twofold: i) the signals of interest can be represented by a vector of latent variables whose dimension is much smaller than that of the original signals and ii) the mapping from the latent space to the observable signal space is given by a succession of layers, each of them consisting of a (learnable) linear transformation followed by a (pre-specified) scalar point-wise nonlinearity. This reduced representation can be later exploited to compress the signals of interest or, alternatively, to solve underdetermined inverse problems, which, after assuming such parsimonious representation, typically become well posed.

To be mathematically rigorous, let ${\mathbf{x}}\in{\mathbb{R}}^{N}$ represent the signal of interest and let ${\mathbf{Z}}\in{\mathbb{R}}^{N_{0}\times F_{0}}$ be a matrix containing the hidden variables with $(N_{0}F_{0})<<N$. The assumption is therefore that the signal ${\mathbf{x}}$ can be written (or well approximated) as

[TABLE]

where $f_{\boldsymbol{\Theta}}(\cdot)$ is a parametric nonlinear function whose parameters are collected in $\boldsymbol{\Theta}$. For the particular case of deep decoders, and with $L$ denoting the total number of layers, the function $f_{\boldsymbol{\Theta}}:{\mathbb{R}}^{N_{0}\times F_{0}}\rightarrow{\mathbb{R}}^{N}$ is assumed to be given as

[TABLE]

where ${\mathbf{Z}}$ is the input, ${\mathbf{x}}$ is the output, $l$ is the layer index, ${\mathcal{T}}_{\boldsymbol{\phi}^{(l)}}^{(l)}\{\cdot\}:{\mathbb{R}}^{N_{l-1}\times F_{l-1}}\to{\mathbb{R}}^{N_{l}\times F_{l}}$ is the linear transformation implemented at layer $l$, $\boldsymbol{\phi}^{(l)}$ are the parameters that define such a transformation, $g^{(l)}:{\mathbb{R}}\rightarrow{\mathbb{R}}$ is a scalar nonlinear operator (possibly different per layer), and ${\mathcal{F}}_{\boldsymbol{\psi}^{(l)}}\{\cdot\}$ is a simple function that re-scales and modifies the mean of each column of $\check{{\mathbf{Y}}}^{(l)}$ using parameters $\boldsymbol{\psi}^{(l)}\in{\mathbb{R}}^{F_{l}\times 2}$. As a side note, such a function is oftentimes not implemented in the last layer. Hence, if $\boldsymbol{\Theta}^{(l)}$ collects the values of $\boldsymbol{\phi}^{(l)}$ and $\boldsymbol{\psi}^{(l)}$ and $\boldsymbol{\Theta}$ is defined as $\boldsymbol{\Theta}:=\{\boldsymbol{\Theta}^{(l)}\}_{l=1}^{L}$, the output ${\mathbf{x}}$ generated by the deep decoder function $f_{\boldsymbol{\Theta}}$ when the input is ${\mathbf{Z}}$ can be effectively computed using (3)-(7).

3 Proposed architecture

The main contribution of this paper is the development of an underparametrized deep decoding architecture for signals supported on a graph ${\mathcal{G}}$. Inspired by [20], which deals with representation of images defined in a 2-D grid, we propose a deep decoding architecture with $L$ layers (see Figure 1 for a pictorial representation) where:

• i)

The input ${\mathbf{Z}}\in{\mathbb{R}}^{N_{0}\times F_{0}}$ is generated as a random realization of a process that is white across rows and columns

• ii)

The linear operator at layer $l$ is defined as

[TABLE]

In the GSP setup considered in this paper, the input feature matrix ${\mathbf{Y}}^{(l-1)}\in{\mathbb{R}}^{N_{l-1}\times F_{l-1}}$ in (8) represents $F_{l-1}$ graph signals, each of them defined on a graph ${\mathcal{G}}_{l-1}$ with $N_{l-1}$ nodes. Similarly, the output matrix ${\hat{\mathbf{Y}}}^{(l)}={\mathcal{T}}_{\boldsymbol{\phi}^{(l)}}^{(l)}\{{\mathbf{Y}}^{(l-1)}\}\in{\mathbb{R}}^{N_{l}\times F_{l}}$ contains $F_{l}$ graph signals, each of them defined on a graph ${\mathcal{G}}_{l}$ with $N_{l}$ nodes. Regarding the particular form of the linear transformation ${\mathcal{T}}_{\boldsymbol{\phi}^{(l)}}^{(l)}$, we note that the (8) postulates a factorized (separated) operation across rows (nodes) and columns (graph signals). In particular, we note that $\boldsymbol{\Phi}^{(l)}\in{\mathbb{R}}^{F_{l}\times F_{l-1}}$ represents a mixing fat matrix that generates new $F_{l}$ features by combining linearly the $F_{l-1}$ features of the previous layer. More importantly for the setup at hand, the tall matrix ${\mathbf{U}}^{(l)}\in{\mathbb{R}}^{N_{l}\times N_{l-1}}$ represents an upsampling operator that maps the values of the graph signal at layer $l-1$ to the graph signal at layer $l$. The design of $\{{\mathbf{U}}^{(l)}\}_{l=1}^{L}$ the subject of the next section.

3.1 Nested collection of upsampling graph-signal operators

While in regular grids the upsampling operator is straightforward, when the signals at hand are defined on irregular domains the problem becomes substantially more challenging. To generate topology-aware upsampling operators, we will rely on agglomerative hierarchical clustering methods [22, 23]. These methods take as input a graph and output a dendrogram, which can be represented as a rooted tree; see Fig. 2. The interpretation of a dendrogram is that of a structure which yields different clusterings at different resolutions. At resolution $\delta=0$ each point (node) is in a cluster of its own. As the resolution parameter $\delta$ increases, nodes start forming clusters. Eventually, the resolutions become coarse enough so that all nodes become members of the same cluster. By cutting the dendrogram at $L+1$ resolutions (including $\delta=0$) we obtain a collection of node sets with parent-child relationships inherited by the refinement of clusters. E.g., the three red nodes in the second graph are the children of the red parent in the coarsest graph in Fig. 2. From these relationships, we define the matrices ${\mathbf{P}}^{(l)}\in\{0,1\}^{N_{l}\times N_{l-1}}$ where the entry $P^{(l)}_{ij}$ is $1$ if node $i$ in layer $l$ is the child of node $j$ in layer $l-1$, and [math] otherwise. Upon defining a (row-normalized) adjacency matrix ${\mathbf{A}}^{(l)}$ between the clusters at layer $l$, the upsampling operator is given by

[TABLE]

where $\gamma$ is a pre-specified constant. Notice that ${\mathbf{U}}^{(l)}$ in (9) first copies the signal value of parents to children (${\mathbf{P}}^{(l)}$) and then every child performs a convex combination between this value and the average signal value of its neighbors. In our numerical experiments, we will test three different ways of defining ${\mathbf{A}}^{(l)}$ for intermediate layers: i) ‘NoA’, where ${\mathbf{A}}^{(l)}={\mathbf{I}}$, ii) ‘Bin’, where two nodes have an unweighted edge if any of their corresponding (grand)children in the original graph ($\delta=0$) have an edge between them, and iii) ‘Wei’, which is a weighted variant of ‘Bin’ depending on the number of edges between (grand)children.

3.2 Deep coding of a graph signal

Suppose now that we are given a signal ${\mathbf{x}}\in{\mathbb{R}}^{N}$ defined on graph ${\mathcal{G}}$ and our goal is to encode ${\mathbf{x}}$ using the architecture defined in (3)-(8) relying on the method to design the upsampling matrices $\{{\mathbf{U}}^{(l)}\}_{l=1}^{L}$ proposed in Section 3.1. Alg. 1 describes the steps to carry out that task. Clearly, Alg. 1 can be used for compression, but also to handle inverse problems such as denoising, as described next.

• •

If compressing ${\mathbf{x}}$ is the ultimate goal, the loss function can be simply set to ${\mathcal{L}}({\mathbf{x}},{\mathbf{x}}^{\prime})\!=\!\|{\mathbf{x}}-{\mathbf{x}}^{\prime}\|_{2}^{2}$ and the matrices $\{\boldsymbol{\Theta}^{(l)}\}_{l=1}^{L}$ obtained after running Alg. 1 are the output of interest, with the resulting compression ratio being $\eta\!=\!(\sum_{l=1}^{L}F_{l}(F_{l-1}+2))/N$.

• •

Suppose now that the observed signal is ${\mathbf{x}}={\mathbf{x}}_{o}+{\mathbf{w}}$ with ${\mathbf{x}}_{o}\in{\mathbb{R}}^{N}$ being the signal of interest and ${\mathbf{w}}\in{\mathbb{R}}^{N}$ being an additive noise of known covariance ${\mathbf{R}}_{ww}$. The goal in this setup is to use the proposed architecture to denoise ${\mathbf{x}}$, so that a better estimate of ${\mathbf{x}}_{o}$ is obtained. In this case, it suffices to run Alg. 1 with ${\mathbf{x}}={\mathbf{x}}_{o}+{\mathbf{w}}$ as input, use the generated output ${\hat{\mathbf{x}}}$ as estimate of ${\mathbf{x}}_{o}$, and set the loss function to ${\mathcal{L}}({\mathbf{x}},{\mathbf{x}}^{\prime})=\|{\mathbf{R}}_{ww}^{-1/2}({\mathbf{x}}-{\mathbf{x}}^{\prime})\|_{2}^{2}$.

4 Numerical experiments

In this section we present experiments to evaluate the performance of the proposed underparametrized deep encoder, showcasing the benefits of including the information of the underlying graph. The code to reproduce the experiments is available on GitHub111https://github.com/reysam93/Graph_Deep_Decoder.

We start by characterizing our architecture using synthetic data. The obtained results are summarized in the 3 panels of Fig. 3. In all three settings, we considered a stochastic block model (SBM) graph with 256 nodes and 4 communities [25]. Edges exist with probability 0.15 if the incident nodes are in the same community and with probability $2.25\times 10^{-3}$ otherwise. The recovery error associated with an observed noisy signal ${\mathbf{x}}={\mathbf{x}}_{o}+{\mathbf{w}}$ is evaluated as $\|{\hat{\mathbf{x}}}-{\mathbf{x}}_{o}\|_{2}^{2}$, with the value reported in the plots being the median error across $100$ generated signals.

Fig. 3(a) plots the denoising performance of 4 architectures with different complexity (i.e., different number of layers and trainable parameters). First and foremost, the results validate our graph-aware approach demonstrating that, in the absence of noise, the approximation error is in all cases below 0.05 ($95\%$ accuracy), even for compression ratios $\eta$ in the order of $25\%$. Moreover, we observe that the most complex model has the smallest error in the absence of noise, but its performance deteriorates quickly as the noise power increases. On the other hand, the simpler architectures are more robust to noise, despite having a larger noiseless error due to their higher compression loss. The architecture’s trade off is then clear: the number of parameters must be large enough to learn a good compressed representation of the signal, but small enough so that the noise is not learnt. With this in mind, the third architecture is specially interesting: it achieves a compression ratio of $25\%$, attains one of the smallest compression errors and is robust to noise. Hence, this will be the architecture used for the remaining simulations.

Fig. 3(b) evaluates the influence of the upsampling operator. The options tested are: (i) ‘GDD-None’: a graph deep decoder model as in (8) but without upsampling (i.e.., ${\mathbf{U}}^{(l)}={\mathbf{I}}$ for all $l$); (ii) ‘DD-Reg’: a deep decoder model with linear upsampling proposed in [20]; (iii) ‘GDD-NoA’; (iv) ‘GDD-Bin’; and (v) ‘GDD-Wei’. The tree last options are graph deep decoder (GDD) schemes where ${\mathbf{U}}^{(l)}$ are constructed as in (9), with $\gamma=0.5$ and ${\mathbf{A}}^{(l)}$ being defined using the methods ‘NoA’, ‘Bin’ and ‘Wei’ presented in Sec. 3.1. It can be seen clearly how methods (i) and (ii), which ignore the structure of the underlying graph, exhibit the worst performance. We also note that, for low noise levels, the error of method (iii) is larger than that of methods (iv) and (v). This is not surprising because the upsampling in (iii) incorporates less information about ${\mathcal{G}}$ than those in (iv) and (v). In addition, to better asses the stability of the results, we computed the relative distances between error percentiles as $\frac{pctl75-pctl25}{median}$. The results for (iii), (iv) and (v) are 0.65, 0.4 and 0.39, respectively, showing that the third method is also more unstable. Finally, because the ‘Wei’ upsampling slightly outperforms the ‘Bin’ upsampling, the former is selected for the remaining tests.

Fig. 3(c) assesses the reconstruction performance for different types of signals. ‘Linear’ signals are created following a linear diffusion process of the form ${\mathbf{x}}_{Linear}=\sum^{T-1}_{t=0}h_{t}{\mathbf{A}}^{t}{\mathbf{s}}$, where ${\mathbf{s}}$ is a sparse signal and $T=6$ [6]. ‘Median’ signals are created from linear signals where the value of each node is the median of its neighbours. In addition, we compare our scheme with a deep decoder with regular upsampling and with a linear graph-aware bandlimited reconstruction as in (1). In the absence of noise the error of the bandlimited model is smaller, and hence, we can assume that both signals are bandlimited. However, when noise is added, the graph deep decoder obtains the best results. It can also be observed that for deep regular decoders the nonlinear transformation entails an increment on the error, while for our architecture it has the opposite effect. This can be explained because the values of the ‘Median’ signals are even more related to the graph topology and, therefore, incorporating the graph information is more relevant.

We close this section by presenting an experiment using the graph and signals from the D&D protein structure database [24], where nodes represent amino acids, links capture their similarity, and signals are their expression level. Using this database, we compare the performance of: (i) a bandlimited recontruction; (ii) an underparametrized deep decoder reconstruction; (iii) an underparametrized graph deep decoder with weighted upsampling reconstruction. Because the minimum size of the graphs is of 50 nodes, an architecture with 48 parameters have been used, instead of the one presented in the previous simulations, achieving an average compression rate of 0.5. The results are shown in Fig. 4, illustrating that the architecture proposed in this paper outperforms the considered alternatives, especially for low noise values.