Active In-Context Learning for Tabular Foundation Models

TL;DR

This paper introduces Tab-AICL, a novel active learning framework for tabular foundation models that enhances sample efficiency by iteratively optimizing the labeled context without retraining, outperforming traditional methods.

Contribution

It formalizes Tab-AICL and proposes four acquisition strategies, demonstrating improved cold-start sample efficiency on 20 benchmarks over gradient-boosting baselines.

Findings

Tab-AICL outperforms gradient-boosting baselines in sample efficiency.

Four acquisition rules are effective in various settings.

Scalable hybrid method reduces computational costs.

Abstract

Active learning (AL) reduces labeling cost by querying informative samples, but in tabular settings its cold-start gains are often limited because uncertainty estimates are unreliable when models are trained on very few labels. Tabular foundation models such as TabPFN provide calibrated probabilistic predictions via in-context learning (ICL), i.e., without task-specific weight updates, enabling an AL regime in which the labeled context - rather than parameters - is iteratively optimized. We formalize Tabular Active In-Context Learning (Tab-AICL) and instantiate it with four acquisition rules: uncertainty (TabPFN-Margin), diversity (TabPFN-Coreset), an uncertainty-diversity hybrid (TabPFN-Hybrid), and a scalable two-stage method (TabPFN-Proxy-Hybrid) that shortlists candidates using a lightweight linear proxy before TabPFN-based selection. Across 20 classification benchmarks, Tab-AICL improves cold-start sample efficiency over retrained gradient-boosting baselines (CatBoost-Margin and XGBoost-Margin), measured by normalized AULC up to 100 labeled samples.