Graph-adaptive Rectified Linear Unit for Graph Neural Networks

TL;DR

This paper introduces GReLU, a topology-aware parametric activation function for GNNs that incorporates neighborhood information into the update stage, enhancing model capacity and performance.

Contribution

The paper proposes GReLU, a novel adaptive activation function that injects topological information into GNN updates, improving their expressiveness and efficiency.

Findings

GReLU improves GNN performance across multiple tasks.

GReLU reduces overfitting and computational costs.

GReLU is compatible with various GNN architectures.

Abstract

Graph Neural Networks (GNNs) have achieved remarkable success by extending traditional convolution to learning on non-Euclidean data. The key to the GNNs is adopting the neural message-passing paradigm with two stages: aggregation and update. The current design of GNNs considers the topology information in the aggregation stage. However, in the updating stage, all nodes share the same updating function. The identical updating function treats each node embedding as i.i.d. random variables and thus ignores the implicit relationships between neighborhoods, which limits the capacity of the GNNs. The updating function is usually implemented with a linear transformation followed by a non-linear activation function. To make the updating function topology-aware, we inject the topological information into the non-linear activation function and propose Graph-adaptive Rectified Linear Unit (GReLU), which is a new parametric activation function incorporating the neighborhood information in a novel and efficient way. The parameters of GReLU are obtained from a hyperfunction based on both node features and the corresponding adjacent matrix. To reduce the risk of overfitting and the computational cost, we decompose the hyperfunction as two independent components for nodes and features respectively. We conduct comprehensive experiments to show that our plug-and-play GReLU method is efficient and effective given different GNN backbones and various downstream tasks.