GATE: How to Keep Out Intrusive Neighbors

TL;DR

GATE enhances Graph Attention Networks by reducing unnecessary neighborhood aggregation, improving performance on heterophilic datasets and enabling deeper architectures without over-smoothing.

Contribution

We introduce GATE, a GAT extension that mitigates over-smoothing, allows deeper networks, and improves performance on heterophilic data by down-weighting irrelevant neighbors.

Findings

GATE reduces over-smoothing in GATs.

GATE outperforms GATs on heterophilic datasets.

Synthetic tests show GATE's ability to control neighborhood aggregation.

Abstract

Graph Attention Networks (GATs) are designed to provide flexible neighborhood aggregation that assigns weights to neighbors according to their importance. In practice, however, GATs are often unable to switch off task-irrelevant neighborhood aggregation, as we show experimentally and analytically. To address this challenge, we propose GATE, a GAT extension that holds three major advantages: i) It alleviates over-smoothing by addressing its root cause of unnecessary neighborhood aggregation. ii) Similarly to perceptrons, it benefits from higher depth as it can still utilize additional layers for (non-)linear feature transformations in case of (nearly) switched-off neighborhood aggregation. iii) By down-weighting connections to unrelated neighbors, it often outperforms GATs on real-world heterophilic datasets. To further validate our claims, we construct a synthetic test bed to analyze a model's ability to utilize the appropriate amount of neighborhood aggregation, which could be of independent interest.