Fast and Robust Contextual Node Representation Learning over Dynamic Graphs

TL;DR

This paper introduces a dynamic graph learning framework based on PPR and sparse attention, improving efficiency and robustness of node representations over evolving graphs, with a new model outperforming baselines especially under noisy conditions.

Contribution

It proposes a unified PPR-based dynamic graph learning framework with theoretical justification, and introduces extsc{GoPPE}, a robust model with improved efficiency and noise resilience.

Findings

 up to 6x efficiency improvement using proximal gradient method

 extsc{GoPPE} outperforms baselines on evolving graphs

 extsc{GoPPE} is robust to noisy initial node attributes

Abstract

Real-world graphs grow rapidly with edge and vertex insertions over time, motivating the problem of efficiently maintaining robust node representation over evolving graphs. Recent efficient GNNs are designed to decouple recursive message passing from the learning process, and favor Personalized PageRank (PPR) as the underlying feature propagation mechanism. However, most PPR-based GNNs are designed for static graphs, and efficient PPR maintenance remains as an open problem. Further, there is surprisingly little theoretical justification for the choice of PPR, despite its impressive empirical performance. In this paper, we are inspired by the recent PPR formulation as an explicit $\ell_1$-regularized optimization problem and propose a unified dynamic graph learning framework based on sparse node-wise attention. We also present a set of desired properties to justify the choice of PPR in STOA GNNs, and serves as the guideline for future node attention designs. Meanwhile, we take advantage of the PPR-equivalent optimization formulation and employ the proximal gradient method (ISTA) to improve the efficiency of PPR-based GNNs upto 6 times. Finally, we instantiate a simple-yet-effective model (\textsc{GoPPE}) with robust positional encodings by maximizing PPR previously used as attention. The model performs comparably to or better than the STOA baselines and greatly outperforms when the initial node attributes are noisy during graph evolution, demonstrating the effectiveness and robustness of \textsc{GoPPE}.