Dynamic Graph Convolutional Networks

TL;DR

This paper introduces two novel neural network architectures that combine LSTM and Graph Convolutional Networks to effectively model dynamic graphs with changing vertices and edges over time, addressing a previously unexplored task.

Contribution

The paper proposes the first architectures that jointly model structured graph data and temporal dynamics using LSTM and GCNs, filling a gap in current methods.

Findings

Achieved promising results on dynamic graph classification tasks.

Demonstrated the effectiveness of combining LSTM with GCNs for temporal graph data.

Showed that the proposed methods outperform baseline approaches.

Abstract

Many different classification tasks need to manage structured data, which are usually modeled as graphs. Moreover, these graphs can be dynamic, meaning that the vertices/edges of each graph may change during time. Our goal is to jointly exploit structured data and temporal information through the use of a neural network model. To the best of our knowledge, this task has not been addressed using these kind of architectures. For this reason, we propose two novel approaches, which combine Long Short-Term Memory networks and Graph Convolutional Networks to learn long short-term dependencies together with graph structure. The quality of our methods is confirmed by the promising results achieved.

Tables

Table 1. Table 1 : Results of the evaluated architectures on semi-supervised classification of sequence of vertices employing the DBLP dataset. We have tested the statistical significance of our result by means of Wilcoxon test , obtaining a p-value < 0.6 % absent percent 0.6 <0.6\% when we have compared WD-GCN and CD-GCN against all the baselines for both the employed scores.

				Accuracy		Unweighted F1 Measure
Network	Hyper-params	Grid		\pbox5cmBest
Config.	\pbox5cmPerformance
mean $\pm$ std		\pbox5cmBest
Config.	\pbox5cmPerformance
mean $\pm$ std
FC+FC	\pbox5cm\nth1 FC nodes:
dropout:	\pbox5cm$\{150,200,250,300,350,400\}$
$\{0\%,10\%,20\%,30\%,40\%,50\%\}$		\pbox5cm250
50%	$49.1\%\pm 1.2\%$		\pbox5cm250
40%	$48.2\%\pm 1.3\%$
\hdashline[1pt/3pt] GC+GC	\pbox5cm\nth1 GC nodes:
dropout:	\pbox5cm$\{150,200,250,300,350,400\}$
$\{0\%,10\%,20\%,30\%,40\%,50\%\}$		\pbox5cm350
50%	$54.8\%\pm 1.4\%$		\pbox5cm350
10%	$54.7\%\pm 1.7\%$
\hdashline[1pt/3pt] LSTM+FC	\pbox5cmLSTM nodes:
dropout:	\pbox5cm$\{100,150,200,300,400\}$
$\{0\%,10\%,20\%,30\%,40\%,50\%\}$		\pbox5cm100
0%	$60.1\%\pm 2.1\%$		\pbox5cm100
0%	$60.4\%\pm 2.3\%$
\hdashline[1pt/3pt] FC+LSTM+FC	\pbox5cmFC nodes:
LSTM nodes:
dropout:	\pbox5cm$\{100,200,300,400\}$
$\{100,200,300,400\}$
$\{0\%,10\%,20\%,30\%,40\%,50\%\}$		\pbox5cm300
300
50%	$61.8\%\pm 1.9\%$		\pbox5cm300
300
50%	$61.8\%\pm 2.4\%$
WD-GCN	\pbox5cmwd-GC nodes:
v-LSTM nodes:
dropout:	\pbox5cm$\{100,200,300,400\}$
$\{100,200,300,400\}$
$\{0\%,10\%,20\%,30\%,40\%,50\%\}$		\pbox5cm300
300
50%	$70.0\%\pm 3.0\%$		\pbox5cm400
300
0%	$70.7\%\pm 2.4\%$
\hdashline[1pt/3pt] CD-GCN	\pbox5cmcd-GC nodes:
v-LSTM nodes:
dropout:	\pbox5cm$\{100,200,300,400\}$
$\{100,200,300,400\}$
$\{0\%,10\%,20\%,30\%,40\%,50\%\}$		\pbox5cm200
100
50%	$70.1\%\pm 2.8\%$		\pbox5cm200
100
50%	$70.5\%\pm 2.7\%$

Table 2. Table 2 : Results of the evaluated architectures on supervised classification of sequence of graphs employing the CAD-120 dataset. CD-GCN is the only technique comparing favourably to all the baselines, resulting in a Wilcoxon test with a p-value lower than 5 % percent 5 5\% for the Unweighted F1 Measure and lower than 10 % percent 10 10\% for the Accuracy.

				Accuracy		Unweighted F1 Measure
Network	Hyper-params	Grid		\pbox5cmBest
Config.	\pbox5cmPerformance
mean $\pm$ std		\pbox5cmBest
Config.	\pbox5cmPerformance
mean $\pm$ std
vs-FC+gs-FC	\pbox5cm\nth1 vs-FC nodes:
dropout:	\pbox5cm$\{100,200,250,300\}$
$\{0\%,20\%,30\%,50\%\}$		\pbox5cm100
20%	$49.9\%\pm 5.2\%$		\pbox5cm200
20%	$48.1\%\pm 7.2\%$
\hdashline[1pt/3pt] GC+gs-FC	\pbox5cm\nth1 GC nodes:
dropout:	\pbox5cm$\{100,200,250,300\}$
$\{0\%,20\%,30\%,50\%\}$		\pbox5cm250
30%	$46.2\%\pm 3.0\%$		\pbox5cm250
50%	$36.7\%\pm 7.9\%$
\hdashline[1pt/3pt] v-LSTM+gs-FC	\pbox5cmLSTM nodes:
dropout:	\pbox5cm$\{100,150,200,300\}$
$\{0\%,20\%,30\%,50\%\}$		\pbox5cm150
0%	$56.8\%\pm 4.1\%$		\pbox5cm150
0%	$53.0\%\pm 9.9\%$
\hdashline[1pt/3pt] vs-FC+v-LSTM+gs-FC	\pbox5cmvs-FC nodes:
v-LSTM nodes:
dropout:	\pbox5cm$\{100,200,250,300\}$
$\{100,150,200,300\}$
$\{0\%,20\%,30\%,50\%\}$		\pbox5cm200
150
20%	$58.7\%\pm 1.5\%$		\pbox5cm200
150
20%	$57.5\%\pm 2.9\%$
WD-GCN	\pbox5cmwd-GC nodes:
v-LSTM nodes:
dropout:	\pbox5cm$\{100,200,250,300\}$
$\{100,150,200,300\}$
$\{0\%,20\%,30\%,50\%\}$		\pbox5cm250
150
30%	$54.3\%\pm 2.6\%$		\pbox5cm250
150
30%	$50.6\%\pm 6.3\%$
\hdashline[1pt/3pt] CD-GCN	\pbox5cmcd-GC nodes:
v-LSTM nodes:
dropout:	\pbox5cm$\{100,200,250,300\}$
$\{100,150,200,300\}$
$\{0\%,20\%,30\%,50\%\}$		\pbox5cm250
150
30%	$60.7\%\pm 8.6\%$		\pbox5cm250
150
30%	$61.0\%\pm 5.3\%$

Equations

$$\stackrel{\mathbf{P}}{\overbracket{\left(\begin{array}{cccc}1& 0& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right)}}\stackrel{\mathbf{v}}{\overbracket{\left(\begin{array}{c}a\\ b\\ c\\ d\end{array}\right)}}=\stackrel{\mathbf{P}\mathbf{v}}{\overbracket{\left(\begin{array}{c}a\\ 0\\ c\\ d\end{array}\right)}}.$$

$${\mathbf{h}}_i={\mathbf{o}}_i\odot \tanh ({\mathbf{c}}_i),$$

$${\mathbf{c}}_i={\mathbf{j}}_i\odot {\tilde{\mathbf{c}}}_i+{\mathbf{f}}_i\odot {\mathbf{c}}_{i-1},$$

$${\mathbf{o}}_i=\sigma ({\mathbf{x}}_i{\mathbf{W}}_o+{\mathbf{h}}_{i-1}{\mathbf{U}}_o+{\mathbf{b}}_o),$$

$${wd-GC}_M:(({\mathbf{X}}_i{)}_{i\in {\mathbb{Z}}_T},({\mathbf{A}}_i{)}_{i\in {\mathbb{Z}}_T})\mapsto (\mathrm{ReLU}({\hat{\mathbf{A}}}_i{\mathbf{X}}_i\mathbf{B}){)}_{i\in {\mathbb{Z}}_T}$$

$${cd-GC}_M:(({\mathbf{X}}_i{)}_{i\in {\mathbb{Z}}_T},({\mathbf{A}}_i{)}_{i\in {\mathbb{Z}}_T})\mapsto ([\mathrm{GF}\mathrm{ReLU}]({\hat{\mathbf{A}}}_i{\mathbf{X}}_i\mathbf{B}){)}_{i\in {\mathbb{Z}}_T}$$

$${v-LSTM}_N:({\mathbf{Z}}_i{)}_{i\in {\mathbb{Z}}_T}\mapsto \left(\begin{array}{c}{\mathrm{LSTM}}_N(({\mathbf{V}}_1^{\prime }{\mathbf{Z}}_i{)}_{i\in {\mathbb{Z}}_T})\\ ⋮\\ {\mathrm{LSTM}}_N(({\mathbf{V}}_L^{\prime }{\mathbf{Z}}_i{)}_{i\in {\mathbb{Z}}_T})\end{array}\right)\in {\mathbb{R}}^{L\times N\times T},$$

$${vs-FC}_k:({\mathbf{Z}}_i{)}_{i\in {\mathbb{Z}}_T}\mapsto (\mathrm{softmax}(\mathbf{W}{\mathbf{Z}}_i+\mathbf{B}){)}_{i\in {\mathbb{Z}}_T}$$

$${gs-FC}_K:({\mathbf{Z}}_i{)}_{i\in {\mathbb{Z}}_T}\mapsto (\mathrm{softmax}({\mathbf{W}}_2\mathrm{ReLU}({\mathbf{W}}_1{\mathbf{Z}}_i+{\mathbf{B}}_1)+{\mathbf{B}}_2){)}_{i\in {\mathbb{Z}}_T}$$

$${v\_wd-GC\_LSTM}_{M,N,k}:$$

$${v\_cd-GC\_LSTM}_{M,N,k}:$$

$$\mathcal{L}=-\sum _{t\in {\mathbb{Z}}_T}\sum _{c\in {\mathbb{Z}}_k}\sum _{v\in {\mathbb{Z}}_{|\mathcal{V}|}}[{\mathbf{Y}}_t{]}_{v,c}\log [{\mathbf{P}}_t^{\text{Lab}}{\mathbf{Z}}_t{]}_{v,c},$$

$${g\_wd-GC\_LSTM}_{M,N,k}:$$

$${g\_cd-GC\_LSTM}_{M,N,k}:$$

$$\mathcal{L}=-\sum _{t\in {\mathbb{Z}}_T}\sum _{c\in {\mathbb{Z}}_k}[{\mathbf{y}}_t{]}_c\log [{\mathbf{z}}_t{]}_c,$$

Full text

1 Introduction

In machine learning, data are usually described as points in a vector space (${\bm{x}}\in\mathbb{R}^{d}$). Nowadays, structured data are ubiquitous and the capability to capture the structural relationships among the points can be particularly useful to improve the effectiveness of the models learned on them.

To this aim, graphs are widely employed to represent this kind of information in terms of nodes/vertices and edges including the local and spatial information arising from data. Consider a $d$-dimensional dataset ${\bm{\mathcal{X}}}=\{{\bm{x}}^{1},\dots,{\bm{x}}^{n}\}\subset\mathbb{R}^{d}$, the graph is extracted from ${\bm{\mathcal{X}}}$ by considering each point as a node and computing the edge weights by means of a function. We obtain a new data representation $\bm{\mathcal{G}}=({\bm{\mathcal{V}}},{\bm{\mathcal{E}}})$, where ${\bm{\mathcal{V}}}$ is a set, which contains vertices, and ${\bm{\mathcal{E}}}$ is a set of weighted pairs of vertices (edges).

Applications to a graph domain can be usually divided into two main categories, called vertex-focused and graph-focused applications. For simplicity of exposition, we just consider the classification problem.111Notice that, the proposed formulation can be trivially rewritten for the regression problem. Under this setting, the vertex-focused applications are characterized by a set of labels ${\bm{\mathcal{L}}}=\{1,\dots,k\}$, a dataset ${\bm{\mathcal{X}}}=\{{\bm{x}}^{1},\dots,{\bm{x}}^{l},{\bm{x}}^{l+1},\dots,{\bm{x}}^{n}\}\subset\mathbb{R}^{d}$, the related graph $\bm{\mathcal{G}}$, and we assume that the first $l$ points ${\bm{x}}^{i}$ (where $1\leq i\leq l$) are labeled and the remaining ${\bm{x}}^{u}$ (where $l+1\leq u\leq n$) are unlabeled. The goal is to classify the unlabeled nodes exploiting the combination of their features and the graph structure by means of a semi-supervised learning approach. Instead, graph-focused applications are related to the goal of learning a function $f$ that maps different graphs to integer values by taking into account the features of the nodes of each graph: $f(\bm{\mathcal{G}}^{i},{\bm{\mathcal{X}}}^{i})\in{\bm{\mathcal{L}}}$. This task can usually be solved using a supervised classification approach on the graph structures.

A number of research works are devoted to classify structured data both for vertex-focused and graph-focused applications [9, 19, 21, 23]. Nevertheless, there is a major limitation in existing studies, most of these research works are focused on static graphs. However, many real-world structured data are dynamic and nodes/edges in the graphs may change during time. In such dynamic scenario, temporal information can also play an important role.

In the last decade, (deep) neural networks have shown their great power and flexibility by learning to represent the world as a nested hierarchy of concepts, achieving outstanding results in many different fields of application. It is important to underline that, just a few research works have been devoted to encode the graph structure directly using a neural network model [1, 3, 4, 12, 15, 20]. Among them, to the best of our knowledge, no one is able to manage dynamic graphs.

To exploit both structured data and temporal information through the use of a neural network model, we propose two novel approaches that combine Long Short Term-Memory networks (LSTMs, [8]) and Graph Convolutional Networks (GCNs, [12]). Both of them are able to deal with vertex-focused applications. These techniques are respectively able to capture temporal information and to properly manage structured data. Furthermore, we have also extended our approaches to deal with graph-focused applications.

LSTMs are a special kind of Recurrent Neural Networks (RNNs, [10]), which are able to improve the learning of long short-term dependencies. All RNNs have the form of a chain of repeating modules of neural networks. Precisely, RNNs are artificial neural networks where connections among units form a directed cycle. This creates an internal state of the network which allows it to exhibit dynamic temporal behavior. In standard RNNs, the repeating module is based on a simple structure, such as a single (hyperbolic tangent) unit. LSTMs extend the repeating module by combining four interacting units.

GCN is a neural network model that directly encodes graph structure, which is trained on a supervised target loss for all the nodes with labels. This approach is able to distribute the gradient information from the supervised loss and to enable it to learn representations exploiting both labeled and unlabeled nodes, thus achieving state-of-the-art results.

The paper is organized as follows: in Section 2 the most related methods are summarized. In Section 3 we describe our approaches. In Section 4 a comparison with baseline methodologies is presented. Section 5 closes the paper by discussing our findings and potential future extensions.

2 Related Work

Many important real-world datasets are in graph form; among all, it is enough to consider: knowledge graphs, social networks, protein-interaction networks, and the World Wide Web.

To deal with this kind of data achieving good classification results, the traditional approaches proposed in literature mainly follow two different directions: to identify structural properties as features to manage them using traditional learning methods, or to propagate the labels to obtain a direct classification.

Zhu et al. [24] propose a semi-supervised learning algorithm based on a Gaussian random field model (also known as Label Propagation). The learning problem is formulated as Gaussian random fields on graphs, where a field is described in terms of harmonic functions, and is efficiently solved using matrix methods or belief propagation. Xu et al. [21] present a semi-supervised factor graph model that is able to exploit the relationships among nodes. In this approach, each vertex is modeled as a variable node and the various relationships are modeled as factor nodes. Grover and Leskovec, in [6], present an efficient and scalable algorithm for feature learning in networks that optimizes a novel network-aware, neighborhood preserving objective function using Stochastic Gradient Descent. Perozzi et al. [18] propose an approach called DeepWalk. This technique uses truncated random walks to efficiently learn representations for vertices in graphs. These latent representations, which encode graph relations in a vector space, can be easily exploited by statistical models thus producing state-of-the-art results.

Unfortunately, the described techniques are not able to deal with graphs that dynamically change in time (nodes/edges in the graphs may change during time). There is a small amount of methodologies that have been designed to classify nodes in dynamic networks [14, 22]. Li et al. [14] propose an approach that is able to learn the latent feature representation and to capture the dynamic patterns. Yao et al. [22] present a Support Vector Machines-based approach that combines the support vectors of the previous temporal instant with the current training data to exploit temporal relationships. Pei et al. [17] define an approach called dynamic Factor Graph Model for node classification in dynamic social networks. More precisely, this approach organizes the dynamic graph data in a sequence of graphs. Three types of factors, called node factor, correlation factor and dynamic factor, are designed to respectively capture node features, node correlations and temporal correlations. Node factor and correlation factor are designed to capture the global and local properties of the graph structures while the dynamic factor exploits the temporal information.

It is important to underline that, very little attention has been devoted to the generalization of neural network models to structured datasets. In the last couple of years, a number of research works have revisited the problem of generalizing neural networks to work on arbitrarily structured graphs [1, 3, 4, 12, 15, 20], some of them achieving promising results in domains that have been previously dominated by other techniques. Scarselli et al. [20] formalize a novel neural network model, called Graph Neural Network (GNNs). This model is based on extending a neural network method with the purpose of processing data in form of graph structures. The GNNs model can process different types of graphs (e.g., acyclic, cyclic, directed, and undirected) and it maps a graph and its nodes into a $D$-dimensional Euclidean space to learn the final classification/regression model. Li et al. [15] extend the GNN model, by relaxing the contractivity requirement of the propagation step through the use of Gated Recurrent Unit [2], and by predicting sequence of outputs from a single input graph. Bruna et al. [1] describe two generalizations of Convolutional Neural Networks (CNNs, [5]). Precisely, the authors propose two variants: one based on a hierarchical clustering of the domain and another based on the spectrum of the Laplacian graph. Duvenaud et al. [4] present another variant of CNNs working on graph structures. This model allows an end-to-end learning on graphs of arbitrary size and shape. Defferrard et al. [3] introduce a formulation of CNNs in the context of spectral graph theory. The model provides efficient numerical schemes to design fast localized convolutional filters on graphs. It is important to notice that, it reaches the same computational complexity of classical CNNs working on any graph structure. Kipf and Welling [12] propose an approach for semi-supervised learning on graph-structured data (GCNs) based on CNNs. In their work, they exploit a localized first-order approximation of the spectral graph convolutions framework [7]. Their model linearly scales in the number of graph edges and learns hidden layer representations encoding local and structural graph features.

Notice that, these neural network architectures are not able to properly deal with temporal information.

3 Our Approaches

In this section, we introduce two novel network architectures to deal with vertex/graph-focused applications. Both of them rely on the following intuitions:

• •

GCNs can effectively deal with graph-structured information, but they lack the ability to handle data structures that change during time. This limitation is (at least) twofold:

(i) inability to manage dynamic vertex features,

(ii) inability to manage dynamic edge connections.

• •

LSTMs excel in finding long short-term dependencies, but they lack the ability to explicitly exploit graph-structured information within it.

Due to the dynamic nature of the tasks we are interested in solving, the new network architectures proposed in this paper will work on ordered sequences of graphs and ordered sequences of vertex features. Notice that, for sequences of length one, this reduces to the vertex/graph-focused applications described in Section 1.

Our contributions are based on the idea of combining an extension of the Graph Convolution (GC, the fundamental layer of the GCNs) and a modified version of LSTM, thus to learn the downstream recurrent units by exploiting both graph structured data and vertex features.

We propose two GC-like layers that take as input a graph sequence and the corresponding ordered sequence of vertex features, and they output an ordered sequence of a new vertex representation. These layers are:

• •

the Waterfall Dynamic-GC layer, which performs at each step of the sequence a graph convolution on the vertex input sequence. An important feature of this layer is that the trainable parameters of each graph convolution are shared among the various step of the sequence;

• •

the Concatenate Dynamic-GC layer, which performs at each step of the sequence a graph convolution on the vertex input features, and concatenates it to the input. Again, the trainable parameters are shared among the steps in the sequence.

Each of the two layers can jointly be used with a modified version of LSTM to perform a semi-supervised classification of sequence of vertices or a supervised classification of sequence of graphs. The difference between the two tasks just consists in how we perform the last processing of the data (for further details, see Equation (1) and Equation (2)).

In the following section we will provide the mathematical definitions of the two modified GC layers, the modified version of LSTM, as well as some other handy definitions that will be useful when we will describe the final network architectures.

3.1 Definitions

Let $(\bm{\mathcal{G}}_{i})_{i\in\mathbb{Z}_{T}}$ with $\mathbb{Z}_{T}\coloneqq\{1,2,\ldots,T\}$ be a finite sequence of undirected graphs $\bm{\mathcal{G}}_{i}=({\bm{\mathcal{V}}}_{i},{\bm{\mathcal{E}}}_{i})$, with ${\bm{\mathcal{V}}}_{i}={\bm{\mathcal{V}}}$ $\forall i\in\mathbb{Z}_{T}$, i.e. all the graphs in the sequence share the same vertices. Considering the graph $\bm{\mathcal{G}}_{i}$, for each vertex $v^{k}\in{\bm{\mathcal{V}}}$ let ${\bm{x}}^{k}_{i}\in\mathbb{R}^{d}$ be the corresponding feature vector. Each step $i$ in the sequence $\mathbb{Z}_{T}$ can completely be defined by its graph $\bm{\mathcal{G}}_{i}$ (modeled by the adjacency matrix222Notice that, the adjacency matrices can be either weighted or unweighted. $\bm{A}_{i}$) and by the vertex-features matrix $\bm{X}_{i}\in\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times d}$ (the matrix whose row vectors are the ${\bm{x}}^{k}_{i}$).

We will denote with $[\bm{Y}]_{i,j}$ the $i$-th row, $j$-th column element of the matrix $\bm{Y}$, and with ${\bm{Y}}^{\prime}$ the transpose of $\bm{Y}$. $\bm{I}_{d}$ is the identity matrix of $\mathbb{R}^{d}$; $\operatorname{softmax}$ and $\operatorname{ReLU}$ are the soft-maximum and the rectified linear unit functions [5].

The matrix $\bm{P}\in\mathbb{R}^{d\times d}$ is a projector on $\mathbb{R}^{d}$ if it is a symmetric, positive semi-definite matrix with $\bm{P}^{2}=\bm{P}$. In particular, it is a diagonal projector if it is a diagonal matrix (with possibly some zero entries on the main diagonal). In other words, a diagonal projector on $\mathbb{R}^{d}$ is diagonal matrix with some $1$s on the main diagonal, that when it is right-multiplied by a $d$-dimensional column vector ${\bm{v}}$ it zeroes out all the entries of ${\bm{v}}$ corresponding to the zeros on the main diagonal of $\bm{P}$:

[TABLE]

We recall here the mathematics of the GC layer [12] and the LSTM [8], since they are the basic building blocks of our contribution. Given a graph with adjacency matrix $\bm{A}\in\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times{|{\bm{\mathcal{V}}}|}}$ and vertex-feature matrix $\bm{X}\in\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times d}$, the GC layer with $M$ output nodes is defined as the function $\operatorname{GC}_{M}:\,\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times d}\times\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times{|{\bm{\mathcal{V}}}|}}\to\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times M}$, such that $\operatorname{GC}_{M}(\bm{X},\bm{A})\coloneqq\operatorname{ReLU}(\hat{\bm{A}}\bm{X}\bm{B})$, where $\bm{B}\in\mathbb{R}^{d\times M}$ is a weight matrix and $\hat{\bm{A}}$ is the re-normalized adjacency matrix, i.e. $\hat{\bm{A}}\coloneqq\tilde{\bm{D}}^{\,\textrm{-}\nicefrac{{1}}{{2}}}\tilde{\bm{A}}\tilde{\bm{D}}^{\,\textrm{-}\nicefrac{{1}}{{2}}}$ with $\tilde{\bm{A}}\coloneqq\bm{A}+\bm{I}_{{|{\bm{\mathcal{V}}}|}}$ and $[\tilde{\bm{D}}]_{kk}\coloneqq\sum_{l}[\tilde{\bm{A}}]_{kl}$.

Given the sequence $({\bm{x}}_{i})_{i\in\mathbb{Z}_{T}}$ with ${\bm{x}}_{i}$ $d$-dimensional row vectors for each $i\in\mathbb{Z}_{T}$, a returning sequence-LSTM with $N$ output nodes, is the function $\operatorname{LSTM}_{N}:\,({\bm{x}}_{i})_{i\in\mathbb{Z}_{T}}\mapsto({\bm{h}}_{i})_{i\in\mathbb{Z}_{T}}$, with ${\bm{h}}_{i}\in\mathbb{R}^{N}$ and

[TABLE]

where $\odot$ is the Hadamard product, $\sigma(x)\coloneqq 1/(1+\mathrm{e}^{\textrm{-}x})$, $\bm{W}_{l}\in\mathbb{R}^{d\times N}$, $\bm{U}_{l}\in\mathbb{R}^{N\times N}$ are weight matrices and ${\bm{b}}_{l}$ are bias vectors, with $l\in\{o,f,j,c\}$.

Definition 1 (wd-GC layer)

Let $(\bm{A}_{i})_{i\in\mathbb{Z}_{T}}$, $(\bm{X}_{i})_{i\in\mathbb{Z}_{T}}$ be, respectively, the sequence of adjacency matrices and the sequence of vertex-feature matrices for the considered graph sequence $(\bm{\mathcal{G}}_{i})_{i\in\mathbb{Z}_{T}}$, with $\bm{A}_{i}\in\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times{|{\bm{\mathcal{V}}}|}}$ and $\bm{X}_{i}\in\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times d}$ $\forall i\in\mathbb{Z}_{T}$. The Waterfall Dynamic-GC layer with $M$ output nodes is the function $\operatorname{wd-GC}_{M}$ with weight matrix $\bm{B}\in\mathbb{R}^{d\times M}$ defined as follows:

[TABLE]

where $\operatorname{ReLU}(\hat{\bm{A}}_{i}\bm{X}_{i}\bm{B})\in\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times M}$, and all the $\hat{\bm{A}}_{i}$ are the re-normalized adjacency matrices of the graph sequence $(\bm{\mathcal{G}}_{i})_{i\in\mathbb{Z}_{T}}$.

The wd-GC layer can be seen as multiple copies of a standard GC layer, all of them sharing the same training weights. Then, the resulting training parameters are $d\cdot M$, independently of the length of the sequence.

In order to introduce the Concatenate Dynamic-GC layer, we recall the definition of the Graph of a Function: considering a function $f$ from $A$ to $B$, $[\operatorname{GF}f]:\,A\to A\times B$, $x\mapsto[\operatorname{GF}f](x)\coloneqq(x,f(x))$. Namely, the $\operatorname{GF}$ operator transforms $f$ into a function returning the concatenation between $x$ and $f(x)$.

Definition 2 (cd-GC layer)

Let $(\bm{A}_{i})_{i\in\mathbb{Z}_{T}}$, $(\bm{X}_{i})_{i\in\mathbb{Z}_{T}}$ be, respectively, the sequence of adjacency matrices and the sequence of vertex-feature matrices for the considered graph sequence $(\bm{\mathcal{G}}_{i})_{i\in\mathbb{Z}_{T}}$, with $\bm{A}_{i}\in\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times{|{\bm{\mathcal{V}}}|}}$ and $\bm{X}_{i}\in\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times d}$ $\forall i\in\mathbb{Z}_{T}$. A Concatenate Dynamic-GC layer with $M$ output nodes is the function $\operatorname{cd-GC}_{M}$ with weight matrix $\bm{B}\in\mathbb{R}^{d\times M}$ defined as follows:

[TABLE]

where $[\operatorname{GF}\operatorname{ReLU}](\hat{\bm{A}}_{i}\bm{X}_{i}\bm{B})\in\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times(M+d)}$, and all the $\hat{\bm{A}}_{i}$ are the re-normalized adjacency matrices of the graph sequence $(\bm{\mathcal{G}}_{i})_{i\in\mathbb{Z}_{T}}$.

Intuitively, cd-GC is a layer made of $T$ copies of GC layers, each copy acting on a specific instant of the sequence. Each output of the $T$ copies is then concatenated with its input, thus resulting in a sequence of graph-convoluted features together with the vertex-features matrix. Note that, the weights $\bm{B}$ are shared among the $T$ copies. The number of learnable parameters of this layer is $d\cdot(d+M)$, independently of the number of steps in the sequence $(\bm{\mathcal{G}}_{i})_{i\in\mathbb{Z}_{T}}$.

Notice that, both the input and the output of wd-GC and cd-GC are sequences of matrices (loosely speaking, third order tensors).

We will now define three additional layers. These will help us in reducing the clutter with the notation when we will introduce in Section 3.2 and Section 3.3 the network architectures we have used to solve the semi-supervised classification of sequence of vertices and the supervised classification of sequence of graphs. Precisely, they are:

(i) the recurrent layer used to process in a parallel fashion the convoluted vertex features,

(ii) the two final layers (one per task) used to map the previous layers outputs into $k$-class probability vectors.

Definition 3 (v-LSTM layer)

Consider $(\bm{Z}_{i})_{i\in\mathbb{Z}_{T}}$ with $\bm{Z}\in\mathbb{R}^{L\times M}$, the Vertex LSTM layer with $N$ output nodes is given by the function $\operatorname{v-LSTM}_{N}$:

[TABLE]

where $\bm{V}_{p}$ is the isometric embedding of $\mathbb{R}$ into $\mathbb{R}^{L}$ defined as $[\bm{V}_{p}]_{i,j}=\delta_{ip}$, and $\delta$ is the Kronecker delta function. The training weights are shared among the $L$ copies of the LSTMs.

Definition 4 (vs-FC layer)

Consider $(\bm{Z}_{i})_{i\in\mathbb{Z}_{T}}$ with $\bm{Z}\in\mathbb{R}^{L\times N}$, the Vertex Sequential Fully Connected layer with $k$ output nodes is given by the function $\operatorname{vs-FC}_{k}$, parameterized by the weight matrix $\bm{W}\in\mathbb{R}^{N\times k}$ and the bias matrix $\mathbb{R}^{L\times k}\ni\bm{B}\coloneqq{({{\bm{b}}}^{\prime},\ldots,{{\bm{b}}}^{\prime})}^{\prime}$:

[TABLE]

with $\operatorname{softmax}(\bm{W}\bm{Z}_{i}+\bm{B})\in\mathbb{R}^{L\times k}$.

Definition 5 (gs-FC layer)

Consider $(\bm{Z}_{i})_{i\in\mathbb{Z}_{T}}$ with $\bm{Z}\in\mathbb{R}^{L\times N}$, the Graph Sequential Fully Connected layer with $k$ output nodes is given by the function $\operatorname{gs-FC}_{k}$, parameterized by the weight matrices $\bm{W}_{1}\in\mathbb{R}^{N\times k}$, $\bm{W}_{2}\in\mathbb{R}^{1\times L}$ and the bias matrices $\mathbb{R}^{L\times k}\ni\bm{B}_{1}\coloneqq{({{\bm{b}}}^{\prime},\ldots,{{\bm{b}}}^{\prime})}^{\prime}$ and $\bm{B}_{2}\in\mathbb{R}^{1\times k}$:

[TABLE]

with $\operatorname{softmax}(\bm{W}_{2}\operatorname{ReLU}(\bm{W}_{1}\bm{Z}_{i}+\bm{B}_{1})+\bm{B}_{2})\in\mathbb{R}^{1\times k}$.

Informally:

(i) the v-LSTM layer acts as $L$ copies of LSTM, each one evaluating the sequence of one row of the input tensor $(\bm{Z}_{i})_{i\in\mathbb{Z}_{T}}$;

(ii) the vs-FC layer acts as $T$ copies of a Fully Connected layer (FC, [5]) with softmax activation, all the copies sharing the parameters. The vs-FC layer outputs $L$ $k$-class probability vectors for each step in the input sequence;

(iii) the gs-FC layer acts as $T$ copies of two FC layers with softmax-ReLU activation, all the copies sharing the parameters. This layer outputs one $k$-class probability vector for each step in the input sequence.

Note that, both the input and the output of vs-FC and v-LSTM are sequences of matrices, while for gs-FC the input is a sequence of matrices and the output is a sequence of vectors.

We have now all the elements to describe our network architectures to address both semi-supervised classification of sequence of vertices and supervised classification of sequence of graphs.

3.2 Semi-Supervised Classification of Sequence of Vertices

Definition 6 (Semi-Supervised Classification of Sequence of Vertices)

Let $(\bm{\mathcal{G}}_{i})_{i\in\mathbb{Z}_{T}}$ be a sequence of $T$ graphs each one made of ${|{\bm{\mathcal{V}}}|}$ vertices, and $(\bm{X}_{i})_{i\in\mathbb{Z}_{T}}$ the related sequence of vertex-features matrices.

Let $(\bm{P}^{\textrm{Lab}}_{i})_{i\in\mathbb{Z}_{T}}$ be a sequence of diagonal projectors on the vector space $\mathbb{R}^{{|{\bm{\mathcal{V}}}|}}$. Define the sequence $(\bm{P}^{\textrm{Unlab}}_{i})_{i\in\mathbb{Z}_{T}}$ by means of $\bm{P}^{\textrm{Unlab}}_{i}\coloneqq\bm{I}_{|{\bm{\mathcal{V}}}|}-\bm{P}^{\textrm{Lab}}_{i}$, $\forall i\in\mathbb{Z}_{T}$; i.e. $\bm{P}^{\textrm{Lab}}_{i}$ and $\bm{P}^{\textrm{Unlab}}_{i}$ identify the labeled and unlabeled vertices of $\bm{\mathcal{G}}_{i}$, respectively. Moreover, let $(\bm{Y}_{i})_{i\in\mathbb{Z}_{T}}$ be a sequence of $T$ matrices with ${|{\bm{\mathcal{V}}}|}$ rows and $k$ columns, satisfying the property $\bm{P}^{\textrm{Lab}}_{i}\bm{Y}_{i}=\bm{Y}_{i}$, where the $j$-th row of the $i$-th matrix represents the one-hot encoding of the $k$-class label of the $j$-th vertex of the $i$-th graph in the sequence, with the $j$-th vertex being a labeled one. Then, semi-supervised classification of sequence of vertices consists in learning a function $f$ such that $\bm{P}^{\textrm{Lab}}_{j}f(\,(\bm{\mathcal{G}}_{i})_{i\in\mathbb{Z}_{T}},(\bm{X}_{i})_{i\in\mathbb{Z}_{T}}\,)_{j}=\bm{Y}_{j}$ and $\bm{P}^{\textrm{Unlab}}_{j}f(\,(\bm{\mathcal{G}}_{i})_{i\in\mathbb{Z}_{T}},\allowbreak(\bm{X}_{i})_{i\in\mathbb{Z}_{T}}\,)_{j}$ is the right labeling for the unlabeled vertices for each $j\in\mathbb{Z}_{T}$.

To address the above task, we propose the networks defined by the following functions:

[TABLE]

where $\circ$ denote the function composition. Both the architectures take $((\bm{X}_{i})_{i\in\mathbb{Z}_{T}},\allowbreak(\bm{A}_{i})_{i\in\mathbb{Z}_{T}})$ as input, and produce a sequence of matrices whose row vectors are the probabilities of each vertex of the graph: $(\bm{Z}_{i})_{i\in\mathbb{Z}_{T}}$ with $\bm{Z}_{i}\in\mathbb{R}^{{|{\bm{\mathcal{V}}}|}\times k}$. For the sake of clarity, in the rest of the paper, we will refer to the networks defined by Equation (1a) and Equation (1b) as Waterfall Dynamic-GCN(WD-GCN) and Concatenate Dynamic-GCN (CD-GCN, see Figure 1(b)), respectively.

Since all the functions involved in the composition are differentiable, the weights of the architectures can be learned using gradient descent methods, employing as loss function the cross-entropy evaluated only on the labeled vertices:

[TABLE]

with the convention that $0\cdot\log 0=0$.

3.3 Supervised Classification of Sequence of Graphs

Definition 7 (Supervised Classification of Sequence of Graphs)

Let $(\bm{\mathcal{G}}_{i})_{i\in\mathbb{Z}_{T}}$ be a sequence of $T$ graphs each one made of ${|{\bm{\mathcal{V}}}|}$ vertices, and $(\bm{X}_{i})_{i\in\mathbb{Z}_{T}}$ the related sequence of vertex-features matrices. Moreover, let $({\bm{y}}_{i})_{i\in\mathbb{Z}_{T}}$ be a sequence of $T$ one-hot encoded $k$-class labels, i.e. ${\bm{y}}_{i}\in\{0,1\}^{k}$. Then, graph-sequence classification task consists in learning a predictive function $f$ such that $f(\,(\bm{\mathcal{G}}_{i})_{i\in\mathbb{Z}_{T}},(\bm{X}_{i})_{i\in\mathbb{Z}_{T}}\,)=({\bm{y}}_{i})_{i\in\mathbb{Z}_{T}}$.

The proposed architectures are defined by the following functions:

[TABLE]

The two architectures take as input $((\bm{X}_{i})_{i\in\mathbb{Z}_{T}},\allowbreak(\bm{A}_{i})_{i\in\mathbb{Z}_{T}})$. The output of wd-GC and cd-GC is processed by a v-LSTM, resulting in a ${|{\bm{\mathcal{V}}}|}\times N$ matrix for each step in the sequence. It is a gs-FC duty to transform this vertex-based prediction into a graph based prediction, i.e. to output a sequence of $k$-class probability vectors $({\bm{z}}_{i})_{i\in\mathbb{Z}_{T}}$. Again, we will use WD-GCN (see Figure 1(a)) and CD-GCN to refer to the networks defined by Equation (2a) and Equation (2b), respectively.

Also under this setting the training can be performed by means of gradient descent methods, with the cross entropy as loss function:

[TABLE]

with the convention $0\cdot\log 0=0$.

4 Experimental Results

In this section we describe the employed datasets, the experimental settings, and the results achieved by our approaches compared with those obtained by baseline methods.

4.1 Datasets

We now present the used datasets. The first one is employed to evaluate our approaches in the context of the vertex-focused applications; instead, the second dataset is used to assess our architectures in the context of the graph-focused applications.

Our first set of data is a subset of DBLP333http://dblp.uni-trier.de/xml/ dataset described in [17]. Conferences from six research communities, including artificial intelligence and machine learning, algorithm and theory, database, data mining, computer vision, and information retrieval, have been considered. Precisely, the co-author relationships from $2001$ to $2010$ are considered and data of each year is organized in a graph form. Each author represents a node in the network and an edge between two nodes exists if two authors have collaborated on a paper in the considered year. Note that, the resulting adjacency matrix is unweighted.

The node features are extracted from each temporal instant using DeepWalk [18] and are composed of 64 values. Furthermore, we have augmented the node features by adding the number of articles published by the authors in each of the six communities, obtaining a features vector composed of $70$ values. This specific task belongs to the vertex-focused applications.

The original dataset is made of $25.215$ authors across the ten years under analysis. Each year $4.786$ authors appear on average, and $114$ authors appear all the years, with an average of $1.594$ authors appearing on two consecutive years.

We have considered the $500$ authors with the highest number of connections during the analyzed $10$ years, i.e. the $500$ vertices among the total $25.215$ with the highest $\sum_{t\in\mathbb{Z}_{10}}\sum_{i}[\bm{A}_{t}]_{i,j}$, with $\bm{A}_{t}$ the adjacency matrix at the $t$-th year. If one of the $500$ selected authors does not appear in the $t$-th year, its feature vector is set to zero.

The final dataset is composed of $10$ vertex-features matrices in $\mathbb{R}^{500\times 70}$, $10$ adjacency matrices belonging to $\mathbb{R}^{500\times 500}$, and each vertex belongs to one of the $6$ classes.

CAD-120444http://pr.cs.cornell.edu/humanactivities/data.php is a dataset composed of $122$ RGB-D videos corresponding to $10$ high-level human activities [13]. Each video is annotated with sub-activity labels, object affordance labels, tracked human skeleton joints and tracked object bounding boxes. The $10$ sub-activity labels are: reaching, moving, pouring, eating, drinking, opening, placing, closing, scrubbing, null. Our second dataset is composed of all the data related to the detection of sub-activities, i.e. no object affordance data have been considered. Notice that, detecting the sub-activities is a challenging problem as it involves complex interactions, since humans can interact with multiple objects during a single activity. This specific task belongs to the graph-focused applications.

Each one of the $10$ high-level activities is characterized by one person, whose $15$ joints are tracked (in position and orientation) in the $3$D space for each frame of the sequence. Moreover, in each high-level activity appears a variable number of objects, for which are registered their bounding boxes in the video frame together with the transformation matrix matching extracted SIFT features [16] from the frame to the ones of the previous frame. Furthermore, there are $19$ objects involved in the videos.

We have built a graph for each video frame: the vertices are the $15$ skeleton joints plus the $19$ objects, while the weighted adjacency matrix has been derived by employing Euclidean distance. Precisely, among two skeleton joints the edge weight is given by the Euclidean distance between their $3$D positions; among two objects it is the $2$D distance between the centroids of their bounding boxes; among an object and a skeleton joint it is the $2$D distance between the centroid of the object bounding box and the skeleton joint projection into the $2$D video frame. All the distances have been scaled between zero and one. When an object does not appear in a frame, its related row and column in the adjacency matrix is set to zero.

Since the videos have different lengths, we have padded all the sequences to match the longest one, which has $1.298$ frames.

Finally, the feature columns have been standardized. The resulting dataset is composed of $122\times 1.298$ vertex-feature matrices belonging to $\mathbb{R}^{34\times 24}$, $122\times 1.298$ adjacency matrices (in $\mathbb{R}^{34\times 34}$), and each graph belongs to one of the $10$ classes.

4.2 Experimental Settings

In our experiments, we have compared the results achieved by the proposed architectures with those obtained by other baseline networks (see Section 4.3 for a full description of the chosen baselines).

For the baselines that are not able to explicitly exploit sequentiality in the data, we have flatten the temporal dimension of all the sequences, thus considering the same point in two different time instants as two different training samples.

The hyper-parameters of all the networks (in terms of number of nodes of each layer and dropout rate) have been appropriately tuned by means of a grid approach. The performances are assessed employing $10$ iterations of Monte Carlo Cross-Validation555This approach randomly selects (without replacement) some fraction of the data to build the training set, and it assignes the rest of the samples to the test set. This process is repeated multiple times, generating (at random) new training and test partitions each time. Notice that, in our experiments, the training set is further split into training and validation. preserving the percentage of samples for each class. It is important to underline that, the $10$ train/test sets are generated once, and they are used to evaluate all the architectures, to keep the experiments as fair as possible. To assess the performances of all the considered architectures we have employed Accuracy and Unweighted F1 Measure666The Unweighted F1 Measure evaluates the F1 scores for each label class, and find their unweighted mean: $\frac{1}{k}\sum_{c\in\mathbb{Z}_{k}}\frac{2p_{c}r_{c}}{p_{c}+r_{c}}$, where $p_{c}$ and $r_{c}$ are the precision and the recall of the class $c$.. Moreover, the training phase has been performed using Adam [11] for a maximum of $100$ epochs, and for each network (independently for Accuracy and F1 Measure) we have selected the epoch where the learned model achieved the best performance on the validation set using the learned model to finally assess the performance on the test set.

4.3 Results

4.3.1 DBLP

We have compared the approaches proposed in Section 3.2 (WD-GCN and CD-GCN) against the following baseline methodologies:

(i) a GCN composed of two layers,

(ii) a network made of two FC layers,

(iii) a network composed of LSTM+FC,

(iv) and a deeper architecture made of FC+LSTM+FC.

Note that, the FC is a Fully Connected layer; when it appears as the first layer of a network it employes a $\operatorname{ReLU}$ activation, instead a $\operatorname{softmax}$ activation is used when it is the last layer of a network.

The test set contains $30\%$ of the $500$ vertices. Moreover, $20\%$ of the remaining vertices (the training ones) have been used for validation purposes. It is important to underline that, an unlabeled vertex remains unlabeled for all the years in the sequence, i.e. considering Definition 6, $\bm{P}^{\textrm{Lab}}_{i}=\bm{P}^{\textrm{Lab}}$, $\forall i\in\mathbb{Z}_{T}$.

In Table 4.3.1, the best hyper-parameter configurations together with the test results of all the evaluated architectures are presented.