Self-Attention Mechanism in Multimodal Context for Banking Transaction Flow

TL;DR

This paper introduces self-attention models, including RNN and Transformer, trained on multimodal banking transaction data to improve categorization and credit risk prediction.

Contribution

It presents a novel application of self-attention mechanisms to multimodal banking transaction data with specialized tokenization and pre-training strategies.

Findings

Transformer-based model outperforms state-of-the-art in transaction categorization.

Pre-trained models improve credit risk prediction accuracy.

Self-attention models effectively handle multimodal BTF data.

Abstract

Banking Transaction Flow (BTF) is a sequential data found in a number of banking activities such as marketing, credit risk or banking fraud. It is a multimodal data composed of three modalities: a date, a numerical value and a wording. We propose in this work an application of self-attention mechanism to the processing of BTFs. We trained two general models on a large amount of BTFs in a self-supervised way: one RNN-based model and one Transformer-based model. We proposed a specific tokenization in order to be able to process BTFs. The performance of these two models was evaluated on two banking downstream tasks: a transaction categorization task and a credit risk task. The results show that fine-tuning these two pre-trained models allowed to perform better than the state-of-the-art approaches for both tasks.