BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classification

TL;DR

BoostLLM introduces a boosting-inspired fine-tuning framework for LLMs that enhances performance in low-data tabular classification by integrating decision-tree paths and residual optimization.

Contribution

This work applies the boosting paradigm to LLM fine-tuning, transforming parameter-efficient training into a multi-round residual process with structured tabular bias incorporation.

Findings

BoostLLM consistently outperforms standard fine-tuning across multiple datasets.

It matches or surpasses XGBoost in low-data regimes.

Scaling BoostLLM with stronger models and longer boosting improves results.

Abstract

Large language models (LLMs) have recently been adapted to tabular prediction by serializing structured features into natural language, but their performance in low-data regimes remains limited compared to gradient-boosted decision trees (GBDTs). In this work, we revisit the boosting paradigm, traditionally associated with tree ensembles, and ask whether it can be applied as a general training principle for LLM fine-tuning. We propose BoostLLM, a framework that transforms parameter-efficient fine-tuning into a multi-round residual optimization process by training sequential PEFT adapters as weak learners. To incorporate tabular inductive bias, BoostLLM integrates decision-tree paths as a second input view alongside raw features; analysis reveals that the path view acts as a structured teacher in early training steps before the model shifts toward feature-driven representations. Empirically, BoostLLM achieves consistent improvements over standard fine-tuning across multiple LLM backbones and datasets, matching or surpassing XGBoost across a wide range of shot counts and outperforming GPT-4o-based methods with a 4B model. We further show that the framework scales: pairing with stronger tree models and extended boosting horizons yields additional gains under appropriate stabilization. These results suggest that boosting can serve as a general training principle for LLM fine-tuning, particularly in low-data regimes for structured data.