Your Spending Needs Attention: Modeling Financial Habits with Transformers

TL;DR

This paper introduces nuFormer, a transformer-based model that leverages self-supervised learning to improve financial transaction data analysis, leading to better customer behavior understanding and recommendation performance without additional data sources.

Contribution

The paper presents a novel transformer-based approach, nuFormer, for self-supervised learning on transaction data, integrating textual and structured features for improved financial modeling.

Findings

Enhanced recommendation accuracy at Nubank.

Representation learning outperforms traditional feature engineering.

No additional data sources needed for performance gains.

Abstract

Predictive models play a crucial role in the financial industry, enabling risk prediction, fraud detection, and personalized recommendations, where slight changes in core model performance can result in billions of dollars in revenue or losses. While financial institutions have access to enormous amounts of user data (e.g., bank transactions, in-app events, and customer support logs), leveraging this data effectively remains challenging due to its complexity and scale. Thus, in many financial institutions, most production models follow traditional machine learning (ML) approaches by converting unstructured data into manually engineered tabular features. Conversely, other domains (e.g., natural language processing) have effectively utilized self-supervised learning (SSL) to learn rich representations from raw data, removing the need for manual feature extraction. In this paper, we investigate using transformer-based representation learning models for transaction data, hypothesizing that these models, trained on massive data, can provide a novel and powerful approach to understanding customer behavior. We propose a new method enabling the use of SSL with transaction data by adapting transformer-based models to handle both textual and structured attributes. Our approach, denoted nuFormer, includes an end-to-end fine-tuning method that integrates user embeddings with existing tabular features. Our experiments demonstrate improvements for large-scale recommendation problems at Nubank. Notably, these gains are achieved solely through enhanced representation learning rather than incorporating new data sources.