A Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformers

TL;DR

This paper introduces a straightforward autoregressive transformer baseline for event prediction in tabular data, outperforming complex existing methods across multiple tasks and use-cases.

Contribution

The work presents a simple, flexible transformer-based approach that generalizes to various event prediction tasks without relying on complex, application-specific techniques.

Findings

Outperforms existing methods on popular datasets

Can predict labels, impute missing data, and model event sequences

Uses standard transformer architecture with elementary positional embeddings

Abstract

Many real-world applications of tabular data involve using historic events to predict properties of new ones, for example whether a credit card transaction is fraudulent or what rating a customer will assign a product on a retail platform. Existing approaches to event prediction include costly, brittle, and application-dependent techniques such as time-aware positional embeddings, learned row and field encodings, and oversampling methods for addressing class imbalance. Moreover, these approaches often assume specific use-cases, for example that we know the labels of all historic events or that we only predict a pre-specified label and not the data's features themselves. In this work, we propose a simple but flexible baseline using standard autoregressive LLM-style transformers with elementary positional embeddings and a causal language modeling objective. Our baseline outperforms existing approaches across popular datasets and can be employed for various use-cases. We demonstrate that the same model can predict labels, impute missing values, or model event sequences.