Mind the truncation gap: challenges of learning on dynamic graphs with recurrent architectures

TL;DR

This paper investigates how the common practice of truncated backpropagation-through-time in graph recurrent neural networks limits their ability to learn long-range dependencies in dynamic graphs, revealing a significant performance gap.

Contribution

The study identifies and analyzes the 'truncation gap' caused by BPTT truncation in GRNNs, highlighting its impact on learning dependencies beyond one hop in CTDGs.

Findings

Full BPTT outperforms truncated BPTT in learning long-range dependencies.

A performance gap exists between full and truncated BPTT in both synthetic and real datasets.

Understanding the truncation gap is crucial for advancing learning on dynamic graphs.

Abstract

Systems characterized by evolving interactions, prevalent in social, financial, and biological domains, are effectively modeled as continuous-time dynamic graphs (CTDGs). To manage the scale and complexity of these graph datasets, machine learning (ML) approaches have become essential. However, CTDGs pose challenges for ML because traditional static graph methods do not naturally account for event timings. Newer approaches, such as graph recurrent neural networks (GRNNs), are inherently time-aware and offer advantages over static methods for CTDGs. However, GRNNs face another issue: the short truncation of backpropagation-through-time (BPTT), whose impact has not been properly examined until now. In this work, we demonstrate that this truncation can limit the learning of dependencies beyond a single hop, resulting in reduced performance. Through experiments on a novel synthetic task and real-world datasets, we reveal a performance gap between full backpropagation-through-time (F-BPTT) and the truncated backpropagation-through-time (T-BPTT) commonly used to train GRNN models. We term this gap the "truncation gap" and argue that understanding and addressing it is essential as the importance of CTDGs grows, discussing potential future directions for research in this area.