Early Stopping Tabular In-Context Learning

TL;DR

This paper introduces an early stopping method for tabular in-context learning that reduces inference time significantly with minimal impact on accuracy, making the process more efficient for large datasets.

Contribution

We propose a dynamic early-stopping technique for in-context learning in tabular models, enabling faster inference without sacrificing performance.

Findings

Inference speed increased by up to 1.3x on small tasks.

Speedup of up to 2.2x on larger tasks.

Negligible performance degradation with early stopping.

Abstract

Tabular foundation models have shown strong performance across various tabular learning tasks via in-context learning, offering robust generalization without any downstream finetuning. However, their inference-time costs remain high, particularly for larger datasets. To address this, we propose early-stopping the in-context learning process. We achieve this by dynamically evaluating whether to stop in-context learning after each Transformer encoder layer. Once stopped, we decode the embedding using a pre-trained layer-wise decoder. Experiments across 34 small classification tasks size show that early stopping in-context learning accelerates inference by up to x1.3 with negligible degradation in predictive performance. To assess scalability, we further evaluate our method on five larger classification tasks, achieving speedups of up to x2.2. Our results demonstrate the potential of early exiting as an effective and practical strategy for improving the efficiency of tabular in-context learning.