Exploring Fine-Tuning for Tabular Foundation Models

TL;DR

This paper systematically evaluates fine-tuning methods for Tabular Foundation Models, revealing that zero-shot performance is strong and fine-tuning benefits depend on specific dataset characteristics, with full supervision sometimes reducing accuracy.

Contribution

It provides the first comprehensive analysis of fine-tuning approaches for TFMs across multiple benchmarks, highlighting when fine-tuning is advantageous or detrimental.

Findings

Zero-shot TFMs perform strongly without fine-tuning.

Fine-tuning benefits vary based on dataset properties.

Full supervised fine-tuning can reduce accuracy or calibration.

Abstract

Tabular Foundation Models (TFMs) have recently shown strong in-context learning capabilities on structured data, achieving zero-shot performance comparable to traditional machine learning methods. We find that zero-shot TFMs already achieve strong performance, while the benefits of fine-tuning are highly model and data-dependent. Meta-learning and PEFT provide moderate gains under specific conditions, whereas full supervised fine-tuning (SFT) often reduces accuracy or calibration quality. This work presents the first comprehensive study of fine-tuning in TFMs across benchmarks including TALENT, OpenML-CC18, and TabZilla. We compare Zero-Shot, Meta-Learning, Supervised (SFT), and parameter-efficient (PEFT) approaches, analyzing how dataset factors such as imbalance, size, and dimensionality affect outcomes. Our findings cover performance, calibration, and fairness, offering practical guidelines on when fine-tuning is most beneficial and its limitations.