Generalizing Graph Convolutional Neural Networks with Edge-Variant Recursions on Graphs

TL;DR

This paper introduces a generalized framework for graph convolutional neural networks using edge-variant recursions, enabling more flexible node-wise weighting of neighbor information and demonstrating superior classification performance.

Contribution

It formulates a unifying framework for GCNNs based on edge-variant filters, offering insights into tradeoffs and guiding the development of new local graph processing methods.

Findings

Superior performance in graph signal classification

Framework unifies and generalizes existing GCNN solutions

Provides guidelines for designing novel local graph filters

Abstract

This paper reviews graph convolutional neural networks (GCNNs) through the lens of edge-variant graph filters. The edge-variant graph filter is a finite order, linear, and local recursion that allows each node, in each iteration, to weigh differently the information of its neighbors. By exploiting this recursion, we formulate a general framework for GCNNs which considers state-of-the-art solutions as particular cases. This framework results useful to i) understand the tradeoff between local detail and the number of parameters of each solution and ii) provide guidelines for developing a myriad of novel approaches that can be implemented locally in the vertex domain. One of such approaches is presented here showing superior performance w.r.t. current alternatives in graph signal classification problems.

Tables

Table 1. TABLE I: Average performance (and std. dev.) for the source localization problem.

Model	Accuracy
Spectral	26.89($\pm$ 0.87)%
Polynomial	74.55($\pm$ 7.32)%
Node Variant (NV) Degree	74.77($\pm$ 7.77)%
Node Variant (NV) S. Proxies	75.62($\pm$ 8.19)%
Edge Variant (EV)	85.47($\pm$10.77)%
Hybrid EV (HEV) Degree	80.53($\pm$10.21)%
Hybrid EV (HEV) S. Proxies	75.37($\pm$ 8.20)%

Table 2. TABLE II: Average performance (and std. dev.) for the authorship attribution problem on text excerpts by Edgar Allan Poe.

Model	Accuracy
Spectral	88.88($\pm$ 1.50)%
Polynomial	79.88($\pm$15.31)%
Node Variant (NV) Degree	88.88($\pm$ 2.62)%
Node Variant (NV) S. Proxies	86.12($\pm$ 5.94)%
Edge Variant (EV)	89.00($\pm$ 2.11)%
Hybrid EV (HEV) Degree	89.18($\pm$ 1.99)%
Hybrid EV (HEV) S. Proxies	90.00($\pm$ 1.21)%

Equations

$$\mathbf{y}=\mathbf{H}(\mathbf{S})\mathbf{x}\text{and}\mathbf{H}(\mathbf{S})=\mathbf{U}h(\mathbf{\Lambda }){\mathbf{U}}^{-1}.$$

$$y_i=\sum _{j\in {\mathcal{N}}_i\cup i}{\varphi }_{i,j}{x}_j$$

$$\begin{array}{rl}\mathbf{y}& ={\varphi }_0{\mathbf{I}}_N\mathbf{x}+{\varphi }_1\mathbf{S}\mathbf{x}+\dots +{\varphi }_K{\mathbf{S}}^K\mathbf{x}\\ & \triangleq {\varphi }_0{\mathbf{w}}^{(0)}+{\varphi }_1{\mathbf{w}}^{(1)}+\dots +{\varphi }_K{\mathbf{w}}^{(K)}.\end{array}$$

$$h(\mathbf{\Lambda })=\sum _{k=0}^K{\varphi }_k{\mathbf{\Lambda }}^k.$$

$${\mathbf{w}}^{(k)}={\mathbf{\Phi }}^{(k)}{\mathbf{w}}^{(k-1)}={\mathbf{\Phi }}^{(k)}{\mathbf{\Phi }}^{(k-1)}\dots {\mathbf{\Phi }}^{(2)}{\mathbf{\Phi }}^{(1)}\mathbf{x}.$$

$$w_i^{(k)}=\sum _{j\in {\mathcal{N}}_i\cup i}{\varphi }_{i,j}^{(k-1)}{w}_j^{(k-1)}.$$

$$\mathbf{y}=\sum _{k=0}^K{\mathbf{\Phi }}^{(k:1)}\mathbf{x}=\stackrel{\mathbf{H}(\mathbf{S})}{\overbrace{\sum _{k=1}^K\left(\prod _{\kappa =1}^k{\mathbf{\Phi }}^{(\kappa )}\right)}}\mathbf{x}.$$

$${\mathbf{z}}_f^l={\sigma }^l\left(\sum _{g=1}^{F_{l-1}}{\mathbf{H}}_{f,g}^l(\mathbf{S}){\mathbf{z}}_g^{l-1}\right)$$

$${\mathbf{H}}_{f,g}^l(\mathbf{S})=\sum _{k=1}^K{\mathbf{\Phi }}_{f,g}^{l,(k:1)}$$

$$\mathbf{H}(\mathbf{S})=\sum _{k=0}^K{\varphi }_k{\mathbf{S}}^k$$

$$h(\mathbf{\Lambda })=\text{diag}(\mathbf{B}\mathbf{b})$$

$$h(\mathbf{\Lambda })=\sum _{k=1}^K\left(\prod _{\kappa =1}^k{\mathbf{\Lambda }}^{(k)}\right).$$

$${\mathbf{C}}_{\mathcal{I}}\text{vec}\left({\mathbf{\Phi }}^{(k)}\right)=0_{|\mathcal{I}|}$$

$$\begin{array}{r}{\mathbf{C}}_{\mathcal{I}}\text{vec}\left(\mathbf{U}{\mathbf{\Lambda }}^{(k)}{\mathbf{U}}^H\right)={\mathbf{C}}_{\mathcal{I}}\text{vec}\left(\mathbf{U}\ast {\mathbf{U}}^H\right){\mathbf{\lambda }}^{(k)}=0_{|\mathcal{I}|}\end{array}$$

$${\mathbf{\lambda }}^{(k)}\in \text{null}\left({\mathbf{C}}_{\mathcal{I}}\text{vec}\left(\mathbf{U}\ast {\mathbf{U}}^H\right)\right).$$

$$h(\mathbf{\Lambda })=\sum _{k=1}^K\left(\prod _{\kappa =1}^k\text{diag}\left({\mathbf{B}}_{\mathbf{U},\mathcal{I}}{\mathbf{\mu }}^{(k)}\right)\right)$$

$$\mathbf{H}(\mathbf{S})=\sum _{k=0}^K\text{diag}\left({\mathbf{C}}_{\mathcal{B}}{\mathbf{\varphi }}_{\mathcal{B}}^{(k)}\right){\mathbf{S}}^k$$

$$\mathbf{H}(\mathbf{S})=\sum _{k=0}^K\left(\prod _{\kappa =0}^k{\mathbf{\Phi }}_{\mathcal{B}}^{(\kappa )}+{\varphi }_k{\mathbf{S}}^k\right)$$

Full text

I Introduction

Graph convolutional neural networks (GCNNs) are gaining momentum as a promising tool for addressing a variety of classification and regression tasks for data that live in irregular spaces, such as network data [1]. With the aim to replicate the success of traditional CNNs, GCNNs play a fundamental role in semi-supervised learning on graphs and classifying different graph signals (i.e., values indexed by the nodes of the graph).

A central aspect of GCNNs is the extension of the convolution operation to graph signals. The seminal work [2] defined convolution as the point-wise multiplication in the graph spectral domain between the projected graph signal in this domain and the transfer function of a learnable filter. Such an approach finds solid grounds on graph signal processing (GSP) theory [3], where the graph spectrum plays the role of the Fourier basis for graph signals. Subsequently, several works exploited this connection and proposed computationally lighter GCNN models. In particular, [4, 5, 6, 7] used the so-called polynomial graph filters [3], while [8, 9] relied on graph filters having a rational transfer function [10]. Differently, [11] designed architectures by using the node-variant (NV) graph filters [12], while [13] introduced MIMO approaches to learn from multiple features.

While the above works introduce techniques that extend CNNs to graphs, they are mostly based on (well-motivated) analogies with classical neural networks. Such a strategy presents, however, its own limitations towards extending these methods to more involved ones that better exploit the graph.

The main aim of this paper is to formulate a general framework that unifies state-of-the-art GCNN architectures facilitating comparison, showing their limitations, and highlighting their potential. For such a goal, it explores the so-called edge-variant (EV) graph filters [14, 15]. The EV graph filter is a local, linear, and finite-order recursion in the node domain where each node weighs differently, in each iteration, the information in its neighborhood. Therefore, it presents the most general linear and local operation that a node can do –gather information from all neighbors and weight each of them differently. This local property puts the EV as a computationally efficient candidate (only local information is exchanged) for capturing detail at the node connection level.

Nevertheless, in the general form, the learnable parameters of the EV graph filter depend on the number of graph edges. To tackle the latter, we first show how state-of-the-art solutions fall under the EV recursion and how they impose parsimonious models on the learnable parameters, rendering their number independent of the graph dimensions. Then, we explore such insights to provide guidelines for designing a variety of novel architectures in the spirit of the EV recursion whose number of parameters is independent of the graph dimensions.

In a nutshell, the contributions of this paper are: $i)$ To formulate a general framework for GCNNs through EV graph filters. $ii)$ To show how the state-of-the-art approaches are specific parameterizations of this EV recursion. $iii)$ To present rigorous design guidelines for GCNNs based on the EV recursion that preserve locality and whose number of parameters is independent of the graph dimension. $iv)$ To introduce one new such an architecture and show its superior performance for graph signal classification tasks.

II Background

II-A Graphs and graph filters

Let ${\mathcal{G}}\!=\!\{{\mathcal{V}},{\mathcal{E}}\}$ be a weighted graph with vertex set ${\mathcal{V}}\!=\!\{1,\ldots,N\}$ of cardinality $|{\mathcal{V}}|\!=\!N$ and edge set ${\mathcal{E}}\subseteq{\mathcal{V}}\times{\mathcal{V}}$ composed of $|{\mathcal{E}}|\!\!=\!\!M$ ordered pairs $(i,j)\!\!\in\!{\mathcal{E}}$ iff there is an edge between nodes $i$ and $j$. For each node $i$, we define the neighborhood set ${\mathcal{N}}_{i}\!=\!\{j\!:\!(j,i)\!\in\!{\mathcal{E}}\}$ as the set of nodes connected to $i$. The sparsity of the edge set of ${\mathcal{G}}$ is represented by an $N\!\times\!N$ matrix ${\mathbf{S}}$, named the graph shift operator matrix, where $[{\mathbf{S}}]_{i,j}\!\neq\!0$ if $(j,i)\!\in\!{\mathcal{E}}$ or $i\!=\!j$. Candidates for ${\mathbf{S}}$ are the graph adjacency ${\mathbf{W}}$, the graph Laplacian matrix (undirected graphs) or any of their normalizations. For generality, in the sequel, we will focus on directed graphs.

Along with ${\mathcal{G}}$, consider a set of signal values (features) ${\mathbf{x}}=[x_{1},\ldots,x_{N}]^{T}\in{\mathbb{R}}^{N}$ in which component $x_{i}$ resides on node $i$. By exploiting the coupling between the graph ${\mathcal{G}}$ and the graph signal ${\mathbf{x}}$, it is possible to compute a graph harmonic analysis for ${\mathbf{x}}$ similarly to the one performed for temporal and image signals. Specifically, given the eigendecomposition ${\mathbf{S}}={\mathbf{U}}\boldsymbol{\Lambda}{\mathbf{U}}^{-1}$, the graph Fourier transform (GFT) of ${\mathbf{x}}$ is ${\hat{\mathbf{x}}}={\mathbf{U}}^{-1}{\mathbf{x}}$. Likewise, the inverse transform is ${\mathbf{x}}={\mathbf{U}}{\hat{\mathbf{x}}}$. Here, ${\mathbf{U}}$ contains along the columns the oscillating modes of the graph and ${\hat{\mathbf{x}}}$ are the respective Fourier coefficients. The eigenvalues $\boldsymbol{\Lambda}$ represent the spectral support for ${\hat{\mathbf{x}}}$ and are commonly referred to as the graph frequencies [3].

Given the Fourier expansion, we can now filter ${\mathbf{x}}$ directly in the spectral domain. That is, for $h:{\mathbb{C}}\to{\mathbb{R}}$ being the filter spectral response (transfer function), the filter output ${\hat{\mathbf{y}}}=h(\boldsymbol{\Lambda}){\hat{\mathbf{x}}}$ is computed as the convolution (pointwise multiplication in the spectral domain) between the filter transfer function $h(\boldsymbol{\Lambda})$ and the GFT of ${\mathbf{x}}$. By means of the inverse GFT, the vertex domain output becomes

[TABLE]

However, (1) is not local, since in computing the output $y_{i}$, node $i$ needs access to the graph signal of non-neighboring nodes. To account for the locality, we can define the filtering operation directly in the vertex domain as the aggregation of neighboring information. The node $i$ output for an order one local filter is

[TABLE]

where the scalar parameter $\phi_{i,j}$ weighs the information of the neighboring node $j$. Nevertheless, this direct vertex domain definition does not enjoy a spectral behavior analysis limiting the connection with the convolution operation.

One way to link the spectral and the vertex domain filtering is to consider the polynomial graph filters [3] with output

[TABLE]

Due to the locality of ${\mathbf{S}}$, ${\mathbf{y}}$ can be obtained in the vertex domain through local information exchange. The state ${\mathbf{w}}^{(0)}$ is simply the graph signal, while ${\mathbf{w}}^{(1)}={\mathbf{S}}{\mathbf{x}}$ consists of one-hop information exchange between adjacent nodes. The higher order states ${\mathbf{w}}^{(k)}={\mathbf{S}}^{k}{\mathbf{x}}$ are computed recursively as ${\mathbf{w}}^{(k)}={\mathbf{S}}{\mathbf{w}}^{(k-1)}$, i.e., by exchanging with the neighbors the previous intermediate state ${\mathbf{w}}^{(k-1)}$. Such an implementation amounts for a complexity of order ${\mathcal{O}}(MK)$. By means of the GFT, the filtering operation in (3) has the transfer function

[TABLE]

Therefore, we conclude that the output ${\mathbf{y}}$ in (3) consists of the convolution between a graph filter with a polynomial transfer function and ${\mathbf{x}}$. This filter enjoys a local implementation and captures detail in a neighborhood of radius $K$ from the node.

II-B Edge-variant graph filters

The edge-variant graph filter is a finite order recursion implemented in the vertex domain in the form similar to (2) [15]. Let $\{\boldsymbol{\Phi}^{(k)}\}_{k=1}^{K}\in{\mathbb{R}}^{N\times N}$ be a collection of $K$ matrices that share the sparsity pattern of ${\mathbf{I}}_{N}+{\mathbf{S}}$. The intermediate states of the EV filter are computed recursively as ${\mathbf{w}}^{(1)}=\boldsymbol{\Phi}^{(1)}{\mathbf{x}}$; ${\mathbf{w}}^{(2)}=\boldsymbol{\Phi}^{(2)}{\mathbf{w}}^{(1)}=\boldsymbol{\Phi}^{(2)}\boldsymbol{\Phi}^{(1)}{\mathbf{x}}$; and

[TABLE]

Since $\boldsymbol{\Phi}^{(1)}$ shares the support with ${\mathbf{S}}+{\mathbf{I}}_{N}$, the state ${\mathbf{w}}^{(1)}$ accounts also for the scaling of ${\mathbf{x}}$ through the diagonal elements (i.e., each node $i$ scales its own signal with a different parameter $\phi_{i,i}^{(1)}$). The higher order states ${\mathbf{w}}^{(k)}$ for $k>1$ are again obtained recursively since the parameter matrices $\boldsymbol{\Phi}^{(k)}$ respect the graph connectivity. Put differently, each $\boldsymbol{\Phi}^{(k)}$ considers a different parameter $\phi_{i,j}^{(k)}$ for each edge $(j,i)\in{\mathcal{E}}$ and adds potential self-loops through $\phi_{i,i}^{(k)}$. Node $i$ computes then the $k$th order state as

[TABLE]

By defining $\boldsymbol{\Phi}^{(k:1)}=\prod_{\kappa=1}^{k}\boldsymbol{\Phi}^{(\kappa)}$ and putting back together all terms in (5), the output of an order $K$ EV graph filter is

[TABLE]

The total number of parameters of the edge-variant graph filter is $K(M+N)$, which is in general smaller than the $N^{2}$ parameters of an arbitrary linear transform ${\mathbf{y}}={\mathbf{H}}{\mathbf{x}}$. In computing the output ${\mathbf{y}}$ in (7) the EV graph filter incurs in an overall computational complexity of order ${\mathcal{O}}(K(M+N))$ which is similar to that of (3) since in general $M\approx N$ or we can consider an EV recursion without self-loops.

The ability to capture local detail at the edge level and the reduced implementation complexity is leveraged next to define graph neural networks (GNN) with a controlled number of parameters and computational complexity matched to the sparsity pattern of the graph.

III Edge-variant graph neural networks

Consider a training set ${\mathcal{T}}=\{{\mathbf{x}}_{i},{\mathbf{z}}_{i}\}_{i=1}^{|{\mathcal{T}}|}$ composed of $|{\mathcal{T}}|$ examples of inputs ${\mathbf{x}}_{i}\in{\mathcal{X}}$ and output representations ${\mathbf{z}}_{i}\in{\mathcal{Z}}$. A GNN leverages the underlying graph representation of the data and learns a model $f(\cdot;{\mathbf{S}}):{\mathcal{X}}\to{\mathcal{Z}}$ such that ${\tilde{\mathbf{z}}}_{i}=f({\mathbf{x}}_{i};{\mathbf{S}})$ minimizes some loss function ${\mathcal{L}}({\tilde{\mathbf{z}}}_{i},{\mathbf{z}}_{i})$ for ${\mathbf{z}}_{i}\in{\mathcal{T}}$ and generalizes well for ${\mathbf{z}}_{i}\notin{\mathcal{T}}$.

To capture several levels of detail, model $f(\cdot,{\mathbf{S}})$ is layered into the cascade of $L$ functions $f(\cdot,{\mathbf{S}})=f^{L}(\cdot,{\mathbf{S}})\circ f^{L-1}(\cdot,{\mathbf{S}})\circ\ldots\circ f^{1}(\cdot,{\mathbf{S}})$ each consisting of a succession of linear transforms and nonlinearities. Layer $l$ produces as output a collection of $F_{l}$ higher level signal features obtained through processing the $F_{l-1}$ features ${\mathbf{z}}_{1}^{l-1},\ldots,{\mathbf{z}}_{F_{l-1}}^{l-1}$ computed at the previous layer. The $f$th higher level feature is computed as

[TABLE]

where $\sigma^{l}(\cdot)$ represents the nonlinearity that might be point-wise (e.g., ReLU) or graph dependent [16] and ${\mathbf{H}}_{f,g}^{l}({\mathbf{S}})$ leverages the graph structure to relate the $g$th input feature ${\mathbf{z}}_{g}^{l-1}$ to the $f$th output feature ${\mathbf{z}}_{f}^{l}$.

The graph basically serves as a parameterization to reduce both the computational complexity and the number of parameters. GCNNs, in particular, consider ${\mathbf{H}}_{f,g}^{l}({\mathbf{S}})$ to be a graph filter that has a spectral interpretation such as (1) or (4). In the sequel, we consider ${\mathbf{H}}_{f,g}^{l}({\mathbf{S}})$ to be an EV graph filter and show that current approaches represent different parameterizations to induce the spectral convolution into the EV GNN.

III-A Properties of the edge-variant neural network layer

From (7) and (8), we build the EV GNN layer as

[TABLE]

characterized by the following properties.

First, it does not require the knowledge of ${\mathbf{S}}$, but only of its support. This is because, differently from current solutions, it will learn from the training data ${\mathcal{T}}$ a collection of $K$ parameter matrices $\boldsymbol{\Phi}^{(k)}$, where each of them acts as a different graph shift operator. Therefore, it represents a robust learning strategy for data residing on graphs whose edge weights are known up to some uncertainty, known only partially, or not known at all, such as biological networks [17].

Second, the computational complexity of each layer is linear in the graph parameters. By setting, $F=\max_{l}F_{l}$, the overall complexity of the EV layer is of order ${\mathcal{O}}(KF^{2}(M+N))$ matching that of current state-of-the-art GCNN approaches.

Third, the number of parameters per layer is, at most $KF^{2}(M+N)$. The latter, although allowing the EV to have the maximum degrees of freedom given a topology, may often be a limitation for large graphs or when $|{\mathcal{T}}|$ is small. Our goal in the next section is, therefore, to show that GCNN layers proposed in the literature are particular cases of (9). Establishing these relationships allows the proposal of novel solutions that increase the descriptive power while preserving an efficient implementation complexity.

III-B Parametrizations

Polynomial GCNN. Several variants of GCNNs introduced in the literature use at each layer graph filters of the form

[TABLE]

where in the filter definition ${\mathbf{H}}({\mathbf{S}})={\mathbf{H}}^{l}_{f,g}({\mathbf{S}})$ we omitted the layer and feature indices to simplify notation. This is the case of [6, 7] which use general polynomials and [4, 5] that consider Chebyshev polynomials.

These filters can be expressed in the form (9) by restricting the parameter matrices to $\boldsymbol{\Phi}^{(k:1)}\!=\!\phi_{k}{\mathbf{S}}^{k}$ for $k\!>\!1$ and $\boldsymbol{\Phi}^{(1)}\!=\!\phi_{0}{\mathbf{I}}_{N}\!+\!\phi_{1}{\mathbf{S}}$. In other words, the EV and the polynomial recursions represent two extremes to implement graph filters locally. The EV recursion allows each node $i$ to learn for each iteration $k$ a different parameter $\phi_{i,j}^{(k)}$ that reflects the importance of $j$’s node features to node $i$. The polynomial implementation instead forces all nodes to weigh the information of all neighbors with the same parameter $\phi_{k}$ within the $k$th iteration. However, this restriction makes the number of parameters $(K+1)F^{2}$ independent from $M$ and $N$.

This way of parameterizing the EV recursion creates now opportunities for proposing a myriad of intermediate solutions that extend (10) towards the edge-variant implementation (9). One such an approach may be a recursion that in addition to (10) considers parameters also for the most critical edges (e.g., edges without which the graph becomes disconnected).

Remark 1: Along with the above works, also [18, 19, 20] and [8, 9] fall under the lines of the polynomial filtering (10). In specific, [18] considers single shifts on graphs using as graph shift operator a learnable weight matrix, [19] considers a Gaussian kernel to mix the neighboring node information, while [20] uses random walks. The works in [8, 9], although aiming to build a GCNN layer by using graph filters with a rational transfer function, approximate the inherited matrix inverse in the vertex domain by a finite iterative algorithm. This finite approximation implicitly transforms these techniques into polynomial recursions whose order depend on the number of iterations (see also [10] for more detail). $\square$

Spectral GCNN. We here establish a link between the edge-variant recursion (9) and the spectral GCNN [2] to provide more insights on its convolutional behavior. The spectral GCNN exploits (1) and learns directly the filter a transfer function $h(\boldsymbol{\Lambda})=\textrm{diag}(h(\lambda_{0}),\ldots,h(\lambda_{N-1}))$. To keep the number of parameters independent from $N$, $h(\boldsymbol{\Lambda})$ is parameterized as

[TABLE]

where ${\mathbf{B}}\in{\mathbb{R}}^{N\times b}$ is a prefixed kernel matrix and ${\mathbf{b}}\in{\mathbb{R}}^{b}$ are the $b\ll N$ learnable parameters. Therefore, the number of parameters for each layer is at most $bF^{2}$, while the computational complexity is of order ${\mathcal{O}}(N^{2})$ required to compute the GFT of the features. Additionally, such an approach requires the eigendecomposition of ${\mathbf{S}}$ (order ${\mathcal{O}}(N^{3})$ to be computed once) and the learned ${\mathbf{H}}({\mathbf{S}})={\mathbf{U}}\textrm{diag}({\mathbf{B}}{\mathbf{b}}){\mathbf{U}}^{-1}$ does not capture the local detail around each vertex.

Nevertheless, this spectral interpretation is useful to understand the EV behavior. We can force the EV recursion (9) to have a spectral response by restricting all coefficient matrices to share the eigenvectors with ${\mathbf{S}}$, i.e., $\boldsymbol{\Phi}^{(k)}={\mathbf{U}}\boldsymbol{\Lambda}^{(k)}{\mathbf{U}}^{-1}$ [15]. Then, EV transfer function becomes

[TABLE]

Subsequently, let ${\mathcal{I}}$ be the index set defining the zero entries of ${\mathbf{S}}+{\mathbf{I}}_{N}$. The fixed support condition for each $\boldsymbol{\Phi}^{(k)}$ is

[TABLE]

where ${\mathbf{C}}_{{\mathcal{I}}}\in\{0,1\}^{|{\mathcal{I}}|\times N^{2}}$ is a selection matrix whose rows are those of ${\mathbf{I}}_{N^{2}}$ indexed by ${\mathcal{I}}$, $\text{vec}(\cdot)$ denotes the vectorization operation, and ${\mathbf{0}}_{|{\mathcal{I}}|}$ is the ${|{\mathcal{I}}|}\times 1$ vector of all zeros. From the properties of the $\text{vec}(\cdot)$ operator, (13) becomes

[TABLE]

where “*” denotes the Khatri-Rao product and $\boldsymbol{\lambda}^{(k)}=\textrm{diag}(\boldsymbol{\Lambda}^{(k)})$ is the $N\times 1$ vector composed by the diagonal elements of $\boldsymbol{\Lambda}^{(k)}$. Put differently, (14) implies

[TABLE]

Finally, by considering ${\mathbf{B}}_{{\mathbf{U}},{\mathcal{I}}}$ as a basis that spans the nullspace of ${\mathbf{C}}_{{\mathcal{I}}}\text{vec}\left({\mathbf{U}}*{\mathbf{U}}^{H}\right)$, we can expand $\boldsymbol{\lambda}^{(k)}$ as $\boldsymbol{\lambda}^{(k)}={\mathbf{B}}_{{\mathbf{U}},{\mathcal{I}}}\boldsymbol{\mu}^{(k)}$ and write (12) as

[TABLE]

for some basis expansion coefficients $\boldsymbol{\mu}^{(k)}$.

The eigendecomposition of $\boldsymbol{\Phi}^{(k)}$ implicitly reduces the total number of layer parameters from $KF^{2}(M+N)$ to $KF^{2}$rank$({\mathbf{B}}_{{\mathbf{U}},{\mathcal{I}}})$ with rank$({\mathbf{B}}_{{\mathbf{U}},{\mathcal{I}}})\ll N$. That is, there is a subclass of the EV recursion that respects the operation of convolution, but, differently from (11), it captures local detail in the vertex domain and enjoys a linear implementation complexity. This subclass has also analogies with (11), which is obtained by setting $K=1$, ${\mathbf{B}}_{{\mathbf{U}},{\mathcal{I}}}={\mathbf{B}}$, and $\boldsymbol{\mu}^{(1)}={\mathbf{b}}$.

In general, we may conclude that the EV recursion implements a GNN layer that goes beyond convolution. Drawing analogies with linear system theory, the GCNN approaches behave as a linear time-invariant (now shift-invariant [3]) filter, while the EV graph filter behaves as linear time-varying (now shift-varying; a different shift per $k$) filter that trades the convolutional interpretation with the ability to capture time-varying (now shift-varying) detail.

Node-variant GCNN. The idea to propose GNNs that extend convolution is also considered in [11], which proposed an architecture having as graph filter the recursion

[TABLE]

where ${\mathcal{B}}\subset{\mathcal{V}}$ is a set of privileged nodes (e.g., the $|{\mathcal{B}}|$ nodes with the highest degree), ${\mathbf{C}}_{\mathcal{B}}\in\{0,1\}^{N\times|{\mathcal{B}}|}$ is a tall binary matrix, and $\boldsymbol{\phi}^{(k)}_{\mathcal{B}}\in{\mathbb{R}}^{|{\mathcal{B}}|}$ is a vector of parameters for the nodes in ${\mathcal{B}}$. In short, (17) learns for each shift, $|{\mathcal{B}}|$ different coefficients for the nodes in ${\mathcal{B}}$ and then maps them through ${\mathbf{C}}_{\mathcal{B}}$ to the remaining nodes ${\mathcal{V}}\backslash{\mathcal{B}}$.

This filter is another way to restrict the EV degrees of freedom, which parameterizes the coefficient matrices to $\boldsymbol{\Phi}^{(1)}\!=\!\textrm{diag}\left({\mathbf{C}}_{{\mathcal{B}}}\boldsymbol{\phi}^{(0)}_{\mathcal{B}}\right)\!+\!\textrm{diag}\left({\mathbf{C}}_{{\mathcal{B}}}\boldsymbol{\phi}^{(1)}_{\mathcal{B}}\right){\mathbf{S}}$ and $\boldsymbol{\Phi}^{(k:1)}\!=\!\textrm{diag}\left({\mathbf{C}}_{{\mathcal{B}}}\boldsymbol{\phi}^{(k)}_{\mathcal{B}}\right){\mathbf{S}}^{k}$ for $k\!>\!1$. That is, (17) is an intermediate approach between the polynomial (10) and the EV (9) recursions and allows each node $i\in{\mathcal{B}}$ to learn, for each $k$, a different parameter $\phi_{i}^{(k)}$ that reflects the importance of all its neighborhood to node $i$. The total number of parameters per layer is at most $(K+1)F^{2}|{\mathcal{B}}|$ while the computational complexity is similar to that of (10).

This different way of parameterizing the EV recursion provides alternative choices to build new intermediate architectures that lever the idea of privileged nodes while giving importance to the edge-based detail. In the sequel, we propose one such an extension that merges insights from the EV, the polynomial, and the NV architecture.

III-C Hybrid edge-variant neural network layer

The hybrid edge-variant (HEV) layer considers the linear operation in (8) to be a graph filter of the form

[TABLE]

where $\boldsymbol{\Phi}^{(0)}_{{\mathcal{B}}}\in{\mathbb{R}}^{N\times N}$ is a diagonal matrix whose $i$th diagonal element $\phi^{(0)}_{i,i}\neq 0$ iff node $i$ belongs to the privileged set ${\mathcal{B}}\subset{\mathcal{V}}$; $\{\boldsymbol{\Phi}_{\mathcal{B}}^{(k)}\}_{k=1}^{K}\in{\mathbb{R}}^{N\times N}$ are a collection of $K$ matrices whose $(i,j)$th element $\phi^{(k)}_{i,j}\neq 0$ iff $i\in{\mathcal{B}}$ and $j\in{\mathcal{N}}_{i}\cup i$; and $\{\phi_{k}\}_{k=0}^{K}$ are a collection of $K+1$ scalars. Put simply, recursion (18) allows nodes in ${\mathcal{B}}$ to learn node-varying parameters for $k=0$ and edge-varying parameters for $k\geq 1$, while the nodes in ${\mathcal{V}}\backslash{\mathcal{B}}$ learn global parameters similar to (10).

This approach represents yet another intermediate architecture between the full convolutional ones and the full EV. By setting $N_{{\mathcal{B}},\text{max}}=\max_{i\in{\mathcal{B}}}|{\mathcal{N}}_{i}|$ as the maximum number of neighbors for the nodes in ${\mathcal{B}}$, the overall number of parameters per layer is at most $F^{2}((K+1)+|{\mathcal{B}}|+K|{\mathcal{B}}|N_{{\mathcal{B}},\text{max}})$. Finally, the HEV implementation cost is of order ${\mathcal{O}}(KF^{2}(2M+N))$.

IV Numerical results

We compare the proposed edge-variant and hybrid edge-variant architectures with the spectral, polynomial, and the node-variant alternatives on a source localization and an author attribution problem. For both experiments, we designed all architectures (except for the spectral GCNN) to have the same computational cost.

IV-A Source Localization

Setup. The goal of this experiment is to find out which community in a stochastic block model (SBM) graph is the source of a diffusion process by observing different diffused signals originated at different (unknown) communities at different (unknown) time instants. ${\mathcal{G}}$ is an undirected SBM graph of $N=50$ nodes divided equally into $C=5$ communities with respective intra- and inter-community edge probabilities of $0.8$ and $0.2$. The initial graph signal $\boldsymbol{\delta}_{i}\in{\mathbb{R}}^{N}$ is a Kronecker delta centered at node $i$ and its realization at time $t$ is ${\mathbf{x}}_{t}={\mathbf{S}}^{t}\boldsymbol{\delta}_{i}$ with ${\mathbf{S}}={\mathbf{W}}/\lambda_{\max}({\mathbf{W}})$. We generated the training set $\{{\mathbf{x}},i\}$ comprising $10,000$ samples by selecting uniformly at random both $i\!\in\!\{1,\ldots,C\}$ and $t\!\in\!\{0,\ldots,N\}$. We then tested the different approaches on $200$ new samples and averaged the performance over $10$ different data and $10$ different graph realizations for an overall of $100$ Monte-Carlo runs.

Models and results. We considered seven architectures each of them composed of the cascade of a graph filtering layer with ReLU nonlinearity and a fully connected layer with softmax nonlinearity. The architectures are: $a)$ a spectral GCNN (11) with ${\mathbf{B}}$ being a cubic spline kernel and $b=5$; $b)$ a polynomial GCNN (10) of order $K=4$; $c)$ two NV GNNs (17) of order $K=4$ and $|{\mathcal{B}}|=5$ privileged nodes selected by $b_{i})$ maximum degree and $b_{ii})$ spectral proxies [21]; $d)$ an EV GNN (9) of order $K=4$; and $e)$ two HEV GNNs (18) of order $K=4$ and $|{\mathcal{B}}|=5$ privileged nodes selected similarly to the NV case. We used the ADAM optimizer with a learning rate $\mu=0.001$ and decaying factors $\beta_{1}=0.9$ and $\beta_{2}=0.999$ run over $20$ epochs with batches of size $100$.

Table I shows the obtained results where we see that because of their increased capacity the EV and the HEV outperform the other alternatives. We observe that the hybrid approaches exploit better the edge-varying part when ${\mathcal{B}}$ is composed of the nodes with the highest degree.

IV-B Authorship Attribution

Setup. In this experiment, we aim to classify if a text excerpt belongs to Edgar Allan Poe or to any other contemporary author. For each text excerpt, we built the graph from the word adjacency network (WAN) between function words that act as nodes. These WANs serve as stylistic signatures for the author (see [22] for full details). Fixed then WAN for Poe, we treat the frequency count of the function words as a graph signal.

In particular, we considered $846$ text excerpts by Poe and randomly split the dataset into $608$ training, $68$ validation, and $170$ testing texts. We sumed the adjacency matrices of the WANs obtained from the $608$ training texts to get the Poe’s signature graph. We completed the training, validation, and test sets by adding respectively other $608$, $68$, and $170$ randomly selected texts by contemporary authors. We set ${\mathbf{S}}$ to be the adjacency matrix of the Poe’s signature graph and averaged the performance over $10$ different data splits.

Models and results. We analyzed the same architectures as in the previous section but set the number of output features to $F=2$, the recursion orders to $K=1$, $b=2$ for the spectral GCNN, and $|{\mathcal{B}}|=2$. We used the same ADAM optimizer for training now over $80$ epochs with batch sizes of $100$ samples.

The results in Table II show that the hybrid approaches offer the best performance highlighting the potential of solutions that consider both edge-dependent and global coefficients. In fact, the polynomial model with global coefficients suffers the most in this experiment.

V Conclusion

We proposed a general framework that unifies $11$ state-of-the-art GCNN architectures into one recursion, named the edge-variant recursion. This unification highlighted the different tradeoff between the number of parameters and the amount of local detail that each approach adopts. Moreover, it shows rigorous ways to choose different tradeoffs and come up with a novel and ad hoc architecture for a problem at hand that is implemented locally in the vertex domain. We here proposed one, among many, extension and showed that it outperforms current solutions for graph signal classification tasks.