PRAGMA: Revolut Foundation Model

TL;DR

PRAGMA is a Transformer-based foundation model trained on large-scale banking event data, enabling improved performance on various financial tasks like credit scoring and fraud detection.

Contribution

It introduces a self-supervised pre-training approach for financial event sequences, creating a versatile representation for multiple downstream banking applications.

Findings

PRAGMA outperforms existing models on multiple financial tasks.

Simple linear models on PRAGMA embeddings achieve strong results.

Lightweight fine-tuning further enhances performance.

Abstract

Modern financial systems generate vast quantities of transactional and event-level data that encode rich economic signals. This paper presents PRAGMA, a family of foundation models for multi-source banking event sequences. Our approach pre-trains a Transformer-based architecture with masked modelling on a large-scale, heterogeneous banking event corpus using a self-supervised objective tailored to the discrete, variable-length nature of financial records. The resulting model supports a wide range of downstream tasks such as credit scoring, fraud detection, and lifetime value prediction: strong performance can be achieved by training a simple linear model on top of the extracted embeddings and can be further improved with lightweight fine-tuning. Through extensive evaluation on downstream tasks, we demonstrate that PRAGMA achieves superior performance across multiple domains directly from raw event sequences, providing a general-purpose representation layer for financial applications.