Integrating Temporal and Structural Context in Graph Transformers for Relational Deep Learning

TL;DR

This paper introduces a novel graph transformer architecture called RGP that effectively integrates temporal and structural information for relational data, enabling multi-task learning and achieving state-of-the-art results.

Contribution

The paper proposes the Relational Graph Perceiver (RGP), a new model that combines a temporal subgraph sampler with a cross-attention-based latent bottleneck for improved relational deep learning.

Findings

RGP achieves state-of-the-art performance on RelBench, SALT, and CTU datasets.

The model effectively captures long-range temporal and structural dependencies.

Supports multi-task learning with disjoint label spaces within a single framework.

Abstract

In domains such as healthcare, finance, and e-commerce, the temporal dynamics of relational data emerge from complex interactions-such as those between patients and providers, or users and products across diverse categories. To be broadly useful, models operating on these data must integrate long-range spatial and temporal dependencies across diverse types of entities, while also supporting multiple predictive tasks. However, existing graph models for relational data primarily focus on spatial structure, treating temporal information merely as a filtering constraint to exclude future events rather than a modeling signal, and are typically designed for single-task prediction. To address these gaps, we introduce a temporal subgraph sampler that enhances global context by retrieving nodes beyond the immediate neighborhood to capture temporally relevant relationships. In addition, we propose the Relational Graph Perceiver (RGP), a graph transformer architecture for relational deep learning that leverages a cross-attention-based latent bottleneck to efficiently integrate information from both structural and temporal contexts. This latent bottleneck integrates signals from different node and edge types into a common latent space, enabling the model to build global context across the entire relational system. RGP also incorporates a flexible cross-attention decoder that supports joint learning across tasks with disjoint label spaces within a single model. Experiments on RelBench, SALT, and CTU show that RGP delivers state-of-the-art performance, offering a general and scalable solution for relational deep learning with support for diverse predictive tasks.