Hierarchical Graph Convolutional Networks for Semi-supervised Node Classification

TL;DR

This paper introduces a deep hierarchical graph convolutional network that enhances global information capture for semi-supervised node classification, outperforming existing methods especially with limited labeled data.

Contribution

The paper proposes a novel deep H-GCN model with a graph coarsening and refinement process to enlarge receptive fields and improve classification accuracy.

Findings

Outperforms state-of-the-art methods by up to 5.9% accuracy.

Shows significant improvements with few labeled samples.

Effectively captures global graph information through hierarchical coarsening.

Abstract

Graph convolutional networks (GCNs) have been successfully applied in node classification tasks of network mining. However, most of these models based on neighborhood aggregation are usually shallow and lack the "graph pooling" mechanism, which prevents the model from obtaining adequate global information. In order to increase the receptive field, we propose a novel deep Hierarchical Graph Convolutional Network (H-GCN) for semi-supervised node classification. H-GCN first repeatedly aggregates structurally similar nodes to hyper-nodes and then refines the coarsened graph to the original to restore the representation for each node. Instead of merely aggregating one- or two-hop neighborhood information, the proposed coarsening procedure enlarges the receptive field for each node, hence more global information can be captured. The proposed H-GCN model shows strong empirical performance on various public benchmark graph datasets, outperforming state-of-the-art methods and acquiring up to 5.9% performance improvement in terms of accuracy. In addition, when only a few labeled samples are provided, our model gains substantial improvements.

Figures

Tables

Table 1. Table 1: Statistics of datasets used in experiments

Dataset	Cora	Citeseer	Pubmed	NELL
Type	Citation network			Knowledge graph
# Vertices	2,708	3,327	19,717	65,755
# Edges	5,429	4,732	44,338	266,144
# Classes	7	6	3	210
# Features	1,433	3,703	500	5,414
Labeling rate	0.052	0.036	0.003	0.003

Table 2. Table 2: Results of node classification in terms of accuracy

Method	Cora	Citeseer	Pubmed	NELL
DeepWalk	67.2%	43.2%	65.3%	58.1%
Planetoid	75.7%	64.7%	77.2%	61.9%
GCN	81.5%	70.3%	79.0%	73.0%
GAT	83.0 ± 0.7%	72.5 ± 0.7%	79.0 ± 0.3%	–
DGCN	83.5%	72.6%	79.3%	74.2%
H-GCN	84.5 ± 0.5%	72.8 ± 0.5%	79.8 ± 0.4%	80.1 ± 0.4%

Table 3. Table 3: Results of node classification in terms of accuracy on Pubmed with labeled vertices varying from 20 per class to 5.

Method	20	15	10	5
GCN	79.0%	76.9%	72.2%	69.0%
GAT	79.0%	77.3%	75.4%	70.3%
DGCN	79.3%	77.4%	76.7%	70.1%
H-GCN	79.8%	79.3%	78.6%	76.5%

Table 4. Table 4: Results of the ablation study

Method	Cora	Citeseer	Pubmed	NELL
H-GCN without coarsen- ing and refining layers	80.3%	70.5%	76.8%	75.9%
H-GCN without node weight embeddings	84.2%	72.4%	79.5%	79.6%
H-GCN	84.5%	72.8%	79.8%	80.1%

Equations

$$G_i=\mathrm{ReLU}\left({\tilde{D}}_i^{-\frac{1}{2}}{\tilde{A}}_i{\tilde{D}}_i^{-\frac{1}{2}}{H}_i{\theta }_i\right),$$

$$s(v_j,v_k)=\frac{A_{jk}}{\sqrt{D(v_j)\cdot D(v_k)}},$$

$$m_{jk}=\left\{\begin{array}[]{ll}1,&\mbox{if $v_{j}$ in $\mathcal{G}_{i}$ is grouped into $v_{k}$ in $\mathcal{G}_{i+1}$};\\ 0,&\mbox{otherwise}.\end{array}\right.$$

$$H_{i+1}=M_i^{\top }\cdot G_i.$$

$$A_{i+1}=M_i^{\top }\cdot A_i\cdot M_i.$$

$$H_i=M_{l-i}\cdot G_i+G_{l-i}.$$

$$G_i=\sum _{j=1}^c{w}_j\cdot G_i^j,$$

$$H_l=\mathrm{softmax}\left(\mathrm{ReLU}\left({\tilde{D}}_l^{-\frac{1}{2}}{\tilde{A}}_l{\tilde{D}}_l^{-\frac{1}{2}}{H}_{l-1}{\theta }_l\right)\right),$$

$$\mathcal{L}=-\sum _{i=1}^m\sum _{y=1}^{|\mathcal{Y}|}\mathbb{I}(h_i=y_i)\log P(h_i,y_i),$$

Full text

1 Introduction

Graphs nowadays become ubiquitous owing to the ability to model complex systems such as social relationships, biological molecules, and publication citations. The problem of classifying graph-structured data is fundamental in many areas. Besides, since there is a tremendous amount of unlabeled data in nature and labeling data is often expensive and time-consuming, it is often challenging and crucial to analyze graphs in a semi-supervised manner. For instance, for semi-supervised node classification in citation networks, where nodes denote articles and edges represent citation, the task is to predict the label of every article with only a few labeled data.

As an efficient and effective approach to graph analysis, network embedding has attracted a lot of research interests. It aims to learn low-dimensional representations for nodes whilst still preserving the topological structure and node feature attributes. Many methods have been proposed for network embedding, which can be used in the node classification task, such as DeepWalk Perozzi et al. (2014) and node2vec Grover and Leskovec (2016). They convert the graph structure into sequences by performing random walks on the graph. Then, the proximity between the nodes can be captured based on the co-occurrence statistics in these sequences. But they are unsupervised algorithms and cannot perform node classification tasks in an end-to-end fashion. Unlike previous random-walk-based approaches, employing neural networks on graphs has been studied extensively in recent years. Using an information diffusion mechanism, the graph neural network (GNN) model updates states of the nodes and propagate them until a stable equilibrium Scarselli et al. (2009). Both of the highly non-linear topological structure and node attributes are fed into the GNN model to obtain the graph embedding. Recently, there is an increasing research interest in applying convolutional operations on the graph. These graph convolutional networks (GCNs) Kipf and Welling (2017); Veličković et al. (2018) are based on the neighborhood aggregation scheme which generates node embedding by combining information from neighborhoods. Comparing with conventional methods, GCNs achieve promising performance in various graph analytical tasks such as node classification and graph classification Defferrard et al. (2016) and has shown effective for many application domains, for instance, recommendation Wu et al. (2019); Cui et al. (2019), traffic forecasting Yu et al. (2018), and action recognition Yan et al. (2018).

Nevertheless, GCN-based models are usually shallow and lack the “graph pooling” mechanism, which restricts the scale of the receptive field. For example, there are only two layers in GCN Kipf and Welling (2017). As each graph convolutional layer acts as the approximation of aggregation on the first-order neighbors, the two-layer GCN model only aggregates information from two-hop neighborhoods for each node. Because of the restricted receptive field, the model has difficulty in obtaining adequate global information. However, it has been observed from the reported results Kipf and Welling (2017) that simply adding more layers will degrade the performance. As explained in Li et al. (2018), each GCN layer acts as a form of Laplacian smoothing in essence, which makes the features of nodes in the same connected component similar. Thereby, adding too many convolutional layers will result in the output features over-smoothed and make them indistinguishable. Meanwhile, deeper neural networks with more parameters are harder to train. Although some recent methods Chen et al. (2018); Xu et al. (2018); Ying et al. (2018) try to get the global information through deeper models, they are either unsupervised models or need many training examples. As a result, they are still not capable of solving the semi-supervised node classification task directly.

To this end, we propose a novel architecture of Hierarchical Graph Convolutional Networks, H-GCN for brevity, for node classification on graphs333To make our results reproducible, all relevant source codes are publicly available at https://github.com/CRIPAC-DIG/H-GCN.. Inspired from the flourish of applying deep architectures and the pooling mechanism into image classification tasks, we design a deep hierarchical model with coarsening mechanisms. The H-GCN model increases the receptive field of graph convolutions and can better capture global information. As illustrated in Figure 1, H-GCN mainly consists of several coarsening layers and refining layers. For each coarsening layer, the graph convolutional operation is first conducted to learn node representations. Then, a coarsening operation is performed to aggregate structurally similar nodes into hyper-nodes. After the coarsening operation, each hyper-node represents a local structure of the original graph, which can facilitate exploiting global structures on the graph. Following coarsening layers, we apply symmetric graph refining layers to restore the original graph structure for node classification tasks. Such a hierarchical model manages to comprehensively capture nodes’ information from local to global perspectives, leading to better node representations.

The main contributions of this paper are twofold. Firstly, to the best of our knowledge, it is the first work to design a deep hierarchical model for the semi-supervised node classification task. Compared to previous work, the proposed model consists of more layers with larger receptive fields, which is able to obtain more global information through the coarsening and refining procedures. Secondly, we conduct extensive experiments on a variety of public datasets and show that the proposed method constantly outperforms other state-of-the-art approaches. Notably, our model gains a considerable improvement over other approaches with very few labeled samples provided for each class.

2 Related Work

In this section, we review some previous work on graph convolutional networks for semi-supervised node classification, hierarchical representation learning on graphs, and graph reduction algorithms.

2.0.1 Graph Convolutional Networks

In the past few years, there has been a surge of applying convolutions on graphs. These approaches are essentially based on the neighborhood aggregation scheme and can be further divided into two branches: spectral approaches and spatial approaches.

The spectral approaches are based on the spectral graph theory to define parameterized filters. Bruna et al. (2014) first define the convolutional operation in the Fourier domain. However, its heavy computational burden limits the application to large-scale graphs. In order to improve efficiency, Defferrard et al. (2016) propose ChebNet to approximate the $K$-polynomial filters by means of a Chebyshev expansion of the graph Laplacian. Kipf and Welling (2017) further simplify the ChebNet by truncating the Chebyshev polynomial to the first-order neighborhood. DGCN Zhuang and Ma (2018) uses random walks to construct a positive mutual information matrix. Then, it utilizes that matrix along with the graph’s adjacency matrix to encode both local consistency and global consistency.

The spatial approaches generate node embedding by combining the neighborhood information in the vertex domain. MoNet Monti et al. (2017) and SplineCNN Fey et al. (2018) integrate the local signals by designing a universe patch operator. To generalize to unseen nodes in an inductive setting, GraphSAGE Hamilton et al. (2017) samples a fixed number of neighbors and employs several aggregation functions, such as concatenation, max-pooling, and LSTM aggregator. GAT Veličković et al. (2018) introduces the attention mechanism to model different influences of neighbors with learnable parameters. Gao et al. (2018) select a fixed number of neighborhood nodes for each feature and enables the use of regular convolutional operations on Euclidean spaces. However, the above two branches of GCNs are usually shallow and cannot obtain adequate global information as a consequence.

2.0.2 Hierarchical Representation Learning on Graphs

Some work has been proposed for learning hierarchical information on graphs. Chen et al. (2018) and Liang et al. (2018) use a coarsening procedure to construct a coarsened graph of smaller size and then employ unsupervised methods, such as DeepWalk Perozzi et al. (2014) and node2vec Grover and Leskovec (2016) to learn node embedding based on that coarsened graph. Then, they conduct a refining procedure to get the original graph embedding. However, their two-stage methods are not capable of utilizing node attribute information and can neither perform node classification task in an end-to-end fashion. JK-Nets Xu et al. (2018) proposes general layer aggregation mechanisms to combine the output representation in every GCN layer. However, it can only propagate information across edges of the graph and are unable to aggregate information hierarchically. Therefore, the hierarchical structure of the graph cannot be learned by JK-Nets. To solve this problem, DiffPool Ying et al. (2018) proposes a pooling layer for graph embedding to reduce the size by a differentiable network. As DiffPool is designed for graph classification tasks, it cannot generate embedding for every node in the graph; hence it cannot be directly applied in node classification scenarios.

2.0.3 Graph Reduction

Many approaches have been proposed to reduce the graph size without losing too much information, which facilitate downstream network analysis tasks such as community discovery and data summarization. There are two main classes of methods that reduce the graph size: graph sampling and graph coarsening. The first category is based on graph sampling strategy Papagelis et al. (2013); Hu and Lau (2013); Chen et al. (2017), which might lose key information during the sampling process. The second category applies graph coarsening strategies that collapse structure-similar nodes into hyper-nodes to generate a series of increasingly coarser graphs. The coarsening operation typically consists of two steps, i.e. grouping and collapsing. At first, every node is assigned to groups in a heuristic manner. Here a group refers to a set of nodes that constitute a hyper-node. Then, these groups are used to generate a coarser graph. For an unmatched node, Hendrickson and Leland (1995) randomly select one of its un-matched neighbors and merge these two nodes. Karypis and Kumar (1998) merge the two un-matched nodes by selecting those with the maximum weight edge. LaSalle and Karypis (2015) use a secondary jump during grouping.

However, these graph reduction approaches are usually used in unsupervised scenarios, such as community detection and graph partition. For semi-supervised node classification tasks, existing graph reduction methods cannot be used directly, as they are not capable of learning complex attributive and structural features of graphs. In this paper, H-GCN conducts graph reduction for non-Euclidean geometry like the pooling mechanism for Euclidean data. In this sense, our work bridges graph reduction for unsupervised tasks to the practical but more challenging semi-supervised node classification problems.

3 The Proposed Method

3.1 Preliminaries

3.1.1 Notations and Problem Definition

For the input undirected graph $\mathcal{G}_{1}=(\mathcal{V}_{1},\mathcal{E}_{1})$, where $\mathcal{V}_{1}$ and $\mathcal{E}_{1}$ are respectively the set of $n_{1}$ nodes and $e_{1}$ edges, let $A_{1}\in\mathbb{R}^{n_{1}\times n_{1}}$ be the adjacency matrix describing its edge weights and $X\in\mathbb{R}^{n_{1}\times d_{1}}$ be the node feature matrix, where $d_{1}$ is the dimension of the attributive features. We use edge weights to indicate connection strengths between nodes. For the H-GCN network, the graph fed into the $i$th layer is represented as $\mathcal{G}_{i}$ with $n_{i}$ nodes. The adjacency matrix and hidden representation matrix of $\mathcal{G}_{i}$ are represented by $A_{i}\in\mathbb{R}^{n_{i}\times n_{i}}$ and $H_{i}\in\mathbb{R}^{n_{i}\times d_{i}}$ respectively.

Since coarsening layers and refining layers are symmetrical, $A_{i}$ is identical to $A_{l-i+1}$, where $l$ is the total number of layers in the network. For example, in the seven-layer model illustrated in Figure 1, $\mathcal{G}_{3}$ is the input graph for the third layer and $\mathcal{G}_{5}$ is the resulting graph from the fourth layer. After one coarsening operation and one refining operation, $\mathcal{G}_{3}$ and $\mathcal{G}_{5}$ share exactly the same topological structure $A_{3}$. As nodes will be assigned as a hyper-node, we define node weight as the number of nodes contained in a hyper-node.

Given the labeled node set $\mathcal{V}_{L}$ containing $m\ll n_{1}$ nodes, where each node $v_{i}\in\mathcal{V}_{L}$ is associated with a label $y_{i}\in\mathcal{Y}$, our objective is to predict labels of $\mathcal{V}\backslash\mathcal{V}_{L}$.

3.1.2 Graph Convolutional Networks

Graph convolutional networks achieve promising generalization in various tasks and our work is built upon the GCN module. At layer $i$, taking graph adjacency matrix $A_{i}$ and hidden representation matrix $H_{i}$ as input, each GCN module outputs a hidden representation matrix $G_{i}\in\mathbb{R}^{n_{i}\times d_{i+1}}$, which is described as:

[TABLE]

where $H_{1}=X$, $\operatorname{ReLU}(x)=\max(0,x)$, adjacency matrix with self-loop $\tilde{A}_{i}=A_{i}+I$, $\tilde{D}_{i}$ is the degree matrix of $\tilde{A}_{i}$, and $\theta_{i}\in\mathbb{R}^{d_{i}\times d_{i+1}}$ is a trainable weight matrix. For ease of parameter tuning, we set output dimension $d_{i}=d$ for all coarsening and refining layers throughout this paper.

3.2 The Overall Architecture

For a H-GCN network of $l$ layers, the $i$th graph coarsening layer first conducts a graph convolutional operation as formulated in Eq. (1) and then aggregates structurally similar nodes into hyper-nodes, producing a coarser graph $\mathcal{G}_{i+1}$ and node embedding matrix $H_{i+1}$ with fewer nodes. The corresponding adjacent matrix $A_{i+1}$ and $H_{i+1}$ will be fed into the $(i+1)$th layer. Symmetrically, the graph refining layer also performs a graph convolution at first and then refines the coarsened graph to restore the finer graph structure. In order to boost optimization in deeper networks, we add shortcut connections He et al. (2016) across each coarsened graph and its corresponding refined part.

Since the topological structure of the graph changes between layers, we further introduce a node weight embedding matrix $S_{i}$, which transforms the number of nodes contained in each hyper-node into real-valued vectors. Both of the node weight embedding and $H_{i}$ will be fed into the $i$th layer. Besides, we add multiple channels by employing different GCNs to explore different feature subspaces.

The graph coarsening layers and refining layers altogether integrate different levels of node features and thus avoid over-smoothing during repeated neighborhood aggregation. After the refining process, we obtain a node embedding matrix $H_{l-1}\in\mathbb{R}^{n_{1}\times d}$, where each row represents a node representation vector. In order to classify each node, we apply an additional GCN module followed by a softmax classifier on $H_{l-1}$.

3.3 The Graph Coarsening Layer

Every graph coarsening layer consists of two steps, i.e. graph convolution and graph coarsening. A GCN module is firstly used to extract structural and attributive features by aggregating neighborhoods’ information as described in Eq. (1). For the graph coarsening procedure, we design the following two hybrid grouping strategies to assign nodes with similar structures into a hyper-node in the coarser graph. We first conduct structural equivalence grouping, followed by structural similarity grouping.

Structural equivalence grouping (SEG).

If two nodes share the same set of neighbors, they are considered to be structurally equivalent. We then assign these two nodes to be a hyper-node. For example, as illustrated in Figure 2, nodes $B$ and $D$ are structurally equivalent, so these two nodes are allocated as a hyper-node. We mark all these structurally equivalent nodes and leave other nodes unmarked to avoid repetitive grouping operation on nodes.

Structural similarity grouping (SSG).

Then, we calculate the structural similarity between the unmarked node pairs $(v_{j},v_{k})$ as the normalized connection strength $s(v_{j},v_{k})$:

[TABLE]

where $A_{jk}$ is the edge weight between $v_{j}$ and $v_{k}$, and $D(\cdot)$ is the node weight.

We iteratively take out an unmarked node $v_{j}$ and calculate normalized connection strengths with all its unmarked neighbors. After that, we select its neighbor node $v_{k}$ which has the largest structural similarity to form a new hyper-node and mark the two nodes. Particularly, if one node is left unmarked and all of its immediate neighbors are marked, it will be marked as well and constitutes a hyper-node by itself. For example, in Figure 2, node pair $(C,E)$ has the largest structural similarity, so they are grouped together to form a hyper-node. After that, since only node $A$ remains unmarked, it constitutes a hyper-node by itself.

Please note that if we take out unmarked nodes in a different order, the resulting hyper-graph will be different. The later we take a node out, the less its neighbors will be left unmarked. So, for a node with fewer neighbors, it has fewer probabilities to be grouped when it is taken out late. Therefore, we take out the unmarked nodes in ascending order according to the number of neighbors.

Using the above two grouping strategies, we are able to acquire all the hyper-nodes. For one hyper-node $v_{i}$, its edge weight to $v_{j}$ is the summation over edge weights of $v_{j}$’s neighbor nodes contained in $v_{i}$. The updated node weights and edge weights will be used in Eq. (2) in the next coarsening layer.

In order to help restore the coarsened graph to original graph, we preserve the grouping relationship between nodes and their corresponding hyper-nodes in a matrix $M_{i}\in\mathbb{R}^{n_{i}\times n_{i+1}}$. Formally, at layer $i$, entry $m_{jk}$ in the grouping matrix $M_{i}$ is calculated as:

[TABLE]

An example of the coarsening operation on a toy graph is given in Figure 2. Note that $m_{11}=1$ in this illustration, since node $A$ constitutes its hyper-node by itself. Next, the hidden node embedding matrix is determined as:

[TABLE]

In the end, we generate a coarser graph $\mathcal{G}_{i+1}$, whose adjacency matrix can be calculated as:

[TABLE]

The coarser graph $\mathcal{G}_{i+1}$ along with the resulting representation matrix $H_{i+1}$ will be fed into the next layer as input. The resulting node embedding to generate in each coarsening layer will then be of lower resolution. The graph coarsening procedure is summarized in Algorithm 1.

3.4 The Graph Refining Layer

To restore the original topological structure of the graph and further facilitate node classification, we stack the same numbers of graph refining layers as coarsening layers. Like the coarsening procedure, each refining layer contains two steps, namely generating node embedding vectors and restoring node representations.

To learn a hierarchical representation of nodes, a GCN is employed at first. Since we have saved the grouping relationship in the grouping matrix during the coarsening process, we utilize $M_{l-i}$ to restore the refined node representation matrix of layer $i$. We further employ residual connections between the two corresponding coarsening and refining layers. In summary, node representations are computed by:

[TABLE]

3.5 Node Weight Embedding and Multiple Channels

As depicted in Figure 2, since different hyper-nodes may have different node weights, we assume such consequent node weights could reflect the hierarchical characteristics of coarsened graphs. In order to better capture the hierarchical information, we use the node weight embedding to supplement information in $H_{i}$. Here we transform the node weight into real-valued vectors by looking up one randomly initialized node weight embedding matrix $V\in\mathbb{R}^{|T|\times p}$, where $T$ is the set of node weights and $p$ is the dimension of the embedding. We apply node weight embedding in every coarsening and refining layer. For graph $\mathcal{G}_{i}$, we obtain its node weight embedding $S_{i}\in\mathbb{R}^{n_{i}\times p}$ by looking up $V$ according to the node weight. For example, if one hyper-node contains three nodes, the third row of $V$ will be selected as its node weight embedding. We then concatenate $H_{i}$ and $S_{i}$ and the resulting $(d+p)$-dimensional matrix will be fed into the next GCN layer subsequently.

Multi-channel mechanisms help explore features in different subspaces and H-GCN employs multiple channels on GCN to obtain rich information jointly at each layer. After obtained $c$ channels $\left[G_{i}^{1},G_{i}^{2},\dots,G_{i}^{c}\right]$, we perform weighted average on these feature maps:

[TABLE]

where $w_{j}$ is a trainable weight of channel $j$.

3.6 The Output Layer

Finally, in the output layer $l$, we use a GCN with a softmax classifier on $H_{l-1}$ to output probabilities of nodes:

[TABLE]

where $\theta_{l}\in\mathbb{R}^{d\times|\mathcal{Y}|}$ is a trainable weight matrix and $H_{l}\in\mathbb{R}^{n_{1}\times|\mathcal{Y}|}$ denotes the probabilities of nodes belonging to each class $y\in\mathcal{Y}$.

The loss function is defined as the cross-entropy of predictions over the labeled nodes:

[TABLE]

where $\mathbb{I}(\cdot)$ is the indicator function, $y_{i}$ is the true label for $v_{i}$, $h_{i}$ is the prediction for labeled node $v_{i}$, and $P(h_{i},y_{i})$ is the predicted probability that $v_{i}$ is of class $y_{i}$.

3.7 Complexity Analysis and Model Comparison

In this section, we analyze the model complexity and compare it with mainstream graph convolutional models, such as GCN and GAT.

For GCN, preprocessing matrices $\tilde{D}_{i}^{-\frac{1}{2}}\tilde{A}_{i}\tilde{D}_{i}^{-\frac{1}{2}}$ takes $O(n^{3})$, and the training process for each layer takes $O(|\mathcal{E}|CF)$, where $\mathcal{E}$ is the edge set and $C,F$ are embedding dimensions. For GAT, the masked attention over all nodes requires $O(n^{2})$ in the training process.

For H-GCN, the preprocessing takes $O(n\log n)$ to sort the unmarked nodes and $O(mn)$ for SSG, where $m$ is the average number of neighborhoods. For training, the complexity is also $O(|\mathcal{E}|CF)$. Therefore, H-GCN is as asymptotically efficient as GCN and is more efficient than GAT.

4 Experiments and Analysis

4.1 Experimental Settings

4.1.1 Datasets

For a comprehensive comparison with state-of-the-art methods, we use four widely-used datasets including three citation networks and one knowledge graph. We conduct semi-supervised node classification task in the transductive setting. The statistics of these datasets are summarized in Table 1. We set the node weight and edge weight of the graph to one for all four datasets. The dataset configuration follows the same setting in Yang et al. (2016); Kipf and Welling (2017) for a fair comparison. For citation networks, documents and citations are treated as nodes and edges, respectively. For the knowledge graph, each triplet $(e_{1},r,e_{2})$ will be assigned with separate relation nodes $r_{1}$ and $r_{2}$ as $(e_{1},r_{1})$ and $(e_{2},r_{2})$, where $e_{1}$ and $e_{2}$ are entities and $r$ is the relation between them. During training, only 20 labels per class are used for each citation network and only one label per class is used for NELL during training. Besides, 500 nodes in each dataset are selected randomly as the validation set. We do not use the labels of the validation set for model training.

4.1.2 Baseline Methods

To evaluate the performance of H-GCN, we compare our method with the following representative methods:

• •

DeepWalk Perozzi et al. (2014) generates the node embedding via random walks in an unsupervised manner, then nodes are classified by feeding the embedding vectors into an SVM classifier.

• •

Planetoid Yang et al. (2016) not only learns node embedding but also predicts the context in graph. It also leverages label information to build both transductive and inductive formulations.

• •

GCN Kipf and Welling (2017) produces node embedding vectors by truncating the Chebyshev polynomial to the first-order neighborhoods.

• •

GAT Veličković et al. (2018) generates node embedding vectors by modeling the differences between the node and its one-hop neighbors.

• •

DGCN Zhuang and Ma (2018) utilizes the graph adjacency matrix and the positive mutual information matrix to encode both local consistency and global consistency.

4.1.3 Parameter Settings

We train our model using Adam optimizer with a learning rate of $0.03$ for $250$ epochs. The dropout is applied to all feature vectors with rates of $0.85$. Besides, the $\ell_{2}$ regularization factor is set to $0.0007$. Considering different scales of datasets, we set the total number of layers $l$ to $9$ for citation networks and $11$ for the knowledge graph, and apply four-channel GCNs in both coarsening and refining layers.

4.2 Node Classification Results

To demonstrate the overall performance of semi-supervised node classification, we compare the proposed method with other state-of-the-art methods. The performance in terms of accuracy is shown in Table 2. The best performance of each column is highlighted in boldface. The performance of our proposed method is reported based on the average of 20 measurements. Note that running GAT on the NELL dataset requires more than 64G memory; hence its performance on NELL is not reported.

The results show that the proposed method consistently outperforms other state-of-the-art methods, which verify the effectiveness of the proposed coarsening and refining mechanisms. Notably, compared with citation networks, H-GCN surpasses other baselines by larger margins on the NELL dataset. To be specific, the accuracy of H-GCN exceeds GCN and DGCN by 7.1% and 5.9% on NELL dataset respectively. We analyze the results as follows.

Regarding traditional random-walk-based algorithms such as DeepWalk and Planetoid, their performance is relatively poor. DeepWalk cannot model the attribute information, which heavily restricts its performance. Though Planetoid combines supervised information with an unsupervised loss, there is information loss of graph structure during random sampling. To avoid that problem, GCN and GAT employ the neighborhood aggregation scheme to boost performance. GAT outperforms GCN as it can model different relations to different neighbors rather than with a pre-defined order. DGCN further jointly models both local and global consistency, yet its global consistency is still obtained through random walks. As a result, the information in the graph structure might lose in DGCN as well. On the contrary, the proposed H-GCN manages to capture global information through different levels of convolutional layers and achieves the best results among all four datasets.

Besides, on the NELL dataset, there are fewer training samples per class than in citation networks. Under such circumstance, training nodes are further away from testing nodes on average. The baseline models with the restricted receptive field are unable to propagate the features and the supervised information of the training nodes to other nodes sufficiently. As a result, the proposed H-GCN with increased receptive fields and deeper layers obtains more promising improvements than baselines.

4.3 Impact of Scale of Training Data

We suppose that a larger receptive field in the convolutional model promotes the propagation of features and labels on graphs. To verify the proposed H-GCN can get a larger receptive field, we reduce the number of training samples to check if H-GCN still performs well when limited labeled data is given. As in nature, there are plenty of unlabeled data; it is also of great significance to train the model with limited labeled data. In this section, we conduct experiments with different numbers of labeled instances on the Pubmed dataset. We vary the number of labeled nodes from 20 to 5 per class, where the labeled data is randomly chosen from the original training set. All parameters are the same as previously described. The corresponding performance in terms of accuracy is reported in Table 3.

From the table, it can be observed that our method outperform other baselines in all cases. With the number of labeled data decreasing, our method obtains a more considerable margin over these baseline algorithms. Especially when only five labeled nodes per class ($\approx$ 0.08% labeling rate) are given, the accuracy of H-GCN exceeds GCN, DGCN, and GAT by 7.5%, 6.4%, and 6.2% respectively. When the number of training data decreases, it is more likely for an unlabeled node to be further away from these labeled nodes. Only when the receptive field is large enough can information from those training nodes be captured. As the receptive field of GCN and GAT does not exceed 2-hop neighborhoods, supervised information contained in the training nodes cannot propagate sufficiently to other nodes. Therefore, these baselines downgrade considerably. However, owing to its larger receptive field, the performance of H-GCN declines slightly when labeled data decreases dramatically. Overall, it is verified that the proposed H-GCN with increased receptive fields is well-suited when training data is extremely scarce and thereby is of significant practical values.

4.4 Ablation Study

To verify the effectiveness of the proposed coarsening and refining layers, we conduct ablation study on coarsening and refining layers and node weight embeddings respectively in this section. The results are shown in Table 4.

Coarsening and refining layers.

We remove all coarsening and refining operations of H-GCN and compare its performance with the original H-GCN. Different from simply adding too many GCN layers, we preserve the short-cut connection between the symmetric layers in the ablation study. From the results, it is evident that the proposed H-GCN has better performance compared to H-GCN without coarsening mechanisms on all datasets. It can be verified that the coarsening and refining mechanisms contribute to the performance improvements since they can obtain global information with larger receptive fields.

Node weight embeddings.

To study the impact of node weight embeddings, we compare H-GCN with no node weight embeddings used. It can be seen from results that the model with node weight embeddings performs better, which verifies the necessity to add this embedding vector in the node embeddings.

4.5 Sensitivity Analysis

Last, we analyze hyper-parameter sensitivity. Specifically, we investigate how different numbers of coarsening layers and different numbers of channels will affect the results respectively. The performance is reported in terms of accuracy on all four datasets. While one parameter studied in the sensitivity analysis is changed, other hyper-parameters remain the same.

Effects of coarsening layers.

Since the coarsening layers in our model control the granularity of the receptive field enlargement, we experiment with one to eight coarsening and symmetric refining layers, where the results are shown in Figure LABEL:sub@fig:coarsening-refining-layers. It is seen that the performance of H-GCN achieves the best when there are four coarsening layers on three citation networks and five on the knowledge graph. It is suspected that, since less labeled nodes are supplied on NELL than others, deeper layers and larger receptive fields are needed. However, when adding too many coarsening layers, the performance drops due to overfitting.

Effects of channel numbers.

Next, we investigate the impact of different amounts of channels on the performance. Multiple channels benefit the graph convolutional network model, since they explore different feature subspaces, as shown in Figure LABEL:sub@fig:channel-numbers. From the figure, it can be found that the performance improves with the number of channels increasing until four channels, which demonstrates that more channels help capture accurate node features. Nevertheless, too many channels will inevitably introduce redundant parameters to the model, leading to overfitting as well.

5 Conclusion

In this paper, we have proposed a novel hierarchical graph convolutional networks for the semi-supervised node classification task. The H-GCN model consists of coarsening layers and symmetric refining layers. By grouping structurally similar nodes to hyper-nodes, our model can get a larger receptive field and enable sufficient information propagation. Compared with other previous work, our proposed H-GCN is deeper and can fully utilize both local and global information. Comprehensive experiments have confirmed that the proposed method consistently outperformed other state-of-the-art methods. In particular, our method has achieved substantial gains over them in the case that labeled data is extremely scarce.

Acknowledgements

This work is jointly supported by National Natural Science Foundation of China (61772528) and National Key Research and Development Program (2016YFB1001000, 2018YFB1402600).