Gradient Derivation for Learnable Parameters in Graph Attention Networks

TL;DR

This paper derives the gradients of learnable parameters in GATv2, providing insights into its training dynamics and addressing inconsistencies in performance across datasets.

Contribution

It offers a comprehensive derivation of parameter gradients for GATv2, enhancing understanding of its training process and potential pitfalls.

Findings

Provides explicit gradient formulas for GATv2 parameters

Highlights factors affecting GATv2 performance variability

Supports future improvements in GAT training methods

Abstract

This work provides a comprehensive derivation of the parameter gradients for GATv2 [4], a widely used implementation of Graph Attention Networks (GATs). GATs have proven to be powerful frameworks for processing graph-structured data and, hence, have been used in a range of applications. However, the achieved performance by these attempts has been found to be inconsistent across different datasets and the reasons for this remains an open research question. As the gradient flow provides valuable insights into the training dynamics of statistically learning models, this work obtains the gradients for the trainable model parameters of GATv2. The gradient derivations supplement the efforts of [2], where potential pitfalls of GATv2 are investigated.