In-context Learning of Evolving Data Streams with Tabular Foundational Models

TL;DR

This paper introduces a novel approach using large tabular transformer models with in-context learning and prompt tuning to improve real-time data stream mining, outperforming traditional ensemble methods in dynamic environments.

Contribution

It presents a new method that leverages pre-trained tabular transformers and in-context learning for adaptive data stream mining, addressing limitations of traditional ensemble algorithms.

Findings

TabPFN with sliding memory outperforms Hoeffding tree ensembles.

Transformers' meta-learning abilities enable better adaptation to drifting data.

The approach achieves consistent improvements across non-stationary benchmarks.

Abstract

State-of-the-art data stream mining has long drawn from ensembles of the Very Fast Decision Tree, a seminal algorithm honored with the 2015 KDD Test-of-Time Award. However, the emergence of large tabular models, i.e., transformers designed for structured numerical data, marks a significant paradigm shift. These models move beyond traditional weight updates, instead employing in-context learning through prompt tuning. By using on-the-fly sketches to summarize unbounded streaming data, one can feed this information into a pre-trained model for efficient processing. This work bridges advancements from both areas, highlighting how transformers' implicit meta-learning abilities, pre-training on drifting natural data, and reliance on context optimization directly address the core challenges of adaptive learning in dynamic environments. Exploring real-time model adaptation, this research demonstrates that TabPFN, coupled with a simple sliding memory strategy, consistently outperforms ensembles of Hoeffding trees, such as Adaptive Random Forest, and Streaming Random Patches, across all non-stationary benchmarks.