Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard

TL;DR

This paper introduces a novel in-context ensemble approach using large language models to dynamically integrate external model predictions for tabular data, achieving expert-level decision-making without raw feature access.

Contribution

The work presents Chain of Tabular Thoughts (CoT$^2$), a prompting strategy enabling LLMs to perform multi-step reasoning for instance-specific model aggregation in tabular prediction.

Findings

Outperforms traditional ensemble methods on various datasets

Enables instance-level adaptive model integration

Leverages LLMs for interpretable, multi-step reasoning

Abstract

Tabular data, a fundamental data format in machine learning, is predominantly utilized in competitions and real-world applications. The performance of tabular models--such as gradient boosted decision trees and neural networks--can vary significantly across datasets due to differences in feature distributions and task characteristics. Achieving top performance on each dataset often requires specialized expert knowledge. To address this variability, practitioners often aggregate the predictions of multiple models. However, conventional aggregation strategies typically rely on static combination rules and lack instance-level adaptability. In this work, we propose an in-context ensemble framework for tabular prediction that leverages large language models (LLMs) to perform dynamic, instance-specific integration of external model predictions. Without access to raw tabular features or semantic information, our method constructs a context around each test instance using its nearest neighbors and the predictions from a pool of external models. Within this enriched context, we introduce Chain of Tabular Thoughts (CoT$^2$), a prompting strategy that guides LLMs through multi-step, interpretable reasoning, making still further progress toward expert-level decision-making. Experimental results show that our method outperforms well-tuned baselines and standard ensemble techniques across a wide range of tabular datasets.