Improving Graph Attention Networks with Large Margin-based Constraints

TL;DR

This paper introduces a margin-based constraint framework for Graph Attention Networks to mitigate over-fitting and over-smoothing, resulting in improved performance on benchmark datasets.

Contribution

The paper proposes a novel margin-based constraint approach for GATs that addresses over-smoothing and over-fitting issues, enhancing their effectiveness.

Findings

Significant performance improvements over state-of-the-art GATs on benchmarks.

Theoretical demonstration of over-smoothing in GATs.

Effective constraints on attention weights and graph structure improve generalization.

Abstract

Graph Attention Networks (GATs) are the state-of-the-art neural architecture for representation learning with graphs. GATs learn attention functions that assign weights to nodes so that different nodes have different influences in the feature aggregation steps. In practice, however, induced attention functions are prone to over-fitting due to the increasing number of parameters and the lack of direct supervision on attention weights. GATs also suffer from over-smoothing at the decision boundary of nodes. Here we propose a framework to address their weaknesses via margin-based constraints on attention during training. We first theoretically demonstrate the over-smoothing behavior of GATs and then develop an approach using constraint on the attention weights according to the class boundary and feature aggregation pattern. Furthermore, to alleviate the over-fitting problem, we propose additional constraints on the graph structure. Extensive experiments and ablation studies on common benchmark datasets demonstrate the effectiveness of our method, which leads to significant improvements over the previous state-of-the-art graph attention methods on all datasets.