Towards Better Understanding Table Instruction Tuning: Decoupling the Effects from Data versus Models

TL;DR

This paper investigates how training data and model architecture individually influence the performance of table instruction tuning in large language models, providing insights into their separate effects and trade-offs.

Contribution

It systematically decouples the effects of training data and model architecture on table instruction tuning performance, offering new insights and state-of-the-art results.

Findings

Achieved new state-of-the-art on Hitab dataset.

Decoupled contributions of data and models through systematic evaluation.

Revealed trade-offs between specialization and generalization in table instruction tuning.

Abstract

Recent advances in natural language processing have leveraged instruction tuning to enhance Large Language Models (LLMs) for table-related tasks. However, previous works train different base models with different training data, lacking an apples-to-apples comparison across the result table LLMs. To address this, we fine-tune base models from the Mistral, OLMo, and Phi families on existing public training datasets. Our replication achieves performance on par with or surpassing existing table LLMs, establishing new state-of-the-art performance on Hitab, a table question-answering dataset. More importantly, through systematic out-of-domain evaluation, we decouple the contributions of training data and the base model, providing insight into their individual impacts. In addition, we assess the effects of table-specific instruction tuning on general-purpose benchmarks, revealing trade-offs between specialization and generalization.