Hierarchical Attention Models for Multi-Relational Graphs

TL;DR

This paper introduces BR-GCN, a novel hierarchical attention-based neural network for multi-relational graphs, improving node classification and link prediction by leveraging bi-level attention mechanisms inspired by transformers and GATs.

Contribution

The paper proposes a new bi-level attention mechanism for GNNs that effectively captures multi-relational data, extending transformer and GAT attention to large-scale heterogeneous graphs.

Findings

BR-GCN outperforms baseline models in node classification by up to 14.95%.

BR-GCN improves link prediction accuracy by up to 7.40%.

Ablation studies confirm the effectiveness of relation-level attention.

Abstract

We present Bi-Level Attention-Based Relational Graph Convolutional Networks (BR-GCN), unique neural network architectures that utilize masked self-attentional layers with relational graph convolutions, to effectively operate on highly multi-relational data. BR-GCN models use bi-level attention to learn node embeddings through (1) node-level attention, and (2) relation-level attention. The node-level self-attentional layers use intra-relational graph interactions to learn relation-specific node embeddings using a weighted aggregation of neighborhood features in a sparse subgraph region. The relation-level self-attentional layers use inter-relational graph interactions to learn the final node embeddings using a weighted aggregation of relation-specific node embeddings. The BR-GCN bi-level attention mechanism extends Transformer-based multiplicative attention from the natural language processing (NLP) domain, and Graph Attention Networks (GAT)-based attention, to large-scale heterogeneous graphs (HGs). On node classification, BR-GCN outperforms baselines from 0.29% to 14.95% as a stand-alone model, and on link prediction, BR-GCN outperforms baselines from 0.02% to 7.40% as an auto-encoder model. We also conduct ablation studies to evaluate the quality of BR-GCN's relation-level attention and discuss how its learning of graph structure may be transferred to enrich other graph neural networks (GNNs). Through various experiments, we show that BR-GCN's attention mechanism is both scalable and more effective in learning compared to state-of-the-art GNNs.