Task-Agnostic Contrastive Pretraining for Relational Deep Learning

TL;DR

This paper introduces a task-agnostic contrastive pretraining method for relational deep learning that captures structural and semantic heterogeneity, enabling scalable, transferable representations across relational databases.

Contribution

It proposes a novel contrastive pretraining framework with three levels of objectives, improving scalability and transferability in relational deep learning models.

Findings

Pretrained models outperform models trained from scratch on benchmarks.

The approach effectively captures relational heterogeneity.

Pretraining enhances model transferability and scalability.

Abstract

Relational Deep Learning (RDL) is an emerging paradigm that leverages Graph Neural Network principles to learn directly from relational databases by representing them as heterogeneous graphs. However, existing RDL models typically rely on task-specific supervised learning, requiring training separate models for each predictive task, which may hamper scalability and reuse. In this work, we propose a novel task-agnostic contrastive pretraining approach for RDL that enables database-wide representation learning. For that aim, we introduce three levels of contrastive objectives$-$row-level, link-level, and context-level$-$designed to capture the structural and semantic heterogeneity inherent to relational data. We implement the respective pretraining approach through a modular RDL architecture and an efficient sampling strategy tailored to the heterogeneous database setting. Our preliminary results on standard RDL benchmarks demonstrate that fine-tuning the pretrained models measurably outperforms training from scratch, validating the promise of the proposed methodology in learning transferable representations for relational data.