A Bird's-Eye Tutorial of Graph Attention Architectures

TL;DR

This paper provides a comprehensive tutorial on graph attention neural networks, systematically analyzing their architectures, strengths, and weaknesses to aid researchers in understanding and applying attention mechanisms in graph-structured data.

Contribution

It offers a structured, bird's-eye view of various graph attention architectures, highlighting their differences and guiding future research in this area.

Findings

Systematic overview of graph attention variants

Analysis of strengths and weaknesses of different architectures

Guidance for researchers on applying attention in GNNs

Abstract

Graph Neural Networks (GNNs) have shown tremendous strides in performance for graph-structured problems especially in the domains of natural language processing, computer vision and recommender systems. Inspired by the success of the transformer architecture, there has been an ever-growing body of work on attention variants of GNNs attempting to advance the state of the art in many of these problems. Incorporating "attention" into graph mining has been viewed as a way to overcome the noisiness, heterogenity and complexity associated with graph-structured data as well as to encode soft-inductive bias. It is hence crucial and advantageous to study these variants from a bird's-eye view to assess their strengths and weaknesses. We provide a systematic and focused tutorial centered around attention based GNNs in a hope to benefit researchers dealing with graph-structured problems. Our tutorial looks at GNN variants from the point of view of the attention function and iteratively builds the reader's understanding of different graph attention variants.