Testing the Limits of Unified Sequence to Sequence LLM Pretraining on Diverse Table Data Tasks

TL;DR

This paper explores the potential of a unified encoder-decoder LLM pretraining approach to handle diverse table data tasks effectively, demonstrating significant performance improvements across multiple scales and tasks.

Contribution

It introduces a shared pretraining methodology for encoder-decoder LLMs on table data, showing scalability from 770M to 11B parameters and benefits of self-supervised objectives.

Findings

Pretraining with self-supervised objectives boosts task performance.

Unified models perform well across diverse table tasks.

Scaling models improves accuracy on table-specific tasks.

Abstract

Tables stored in databases and tables which are present in web pages and articles account for a large part of semi-structured data that is available on the internet. It then becomes pertinent to develop a modeling approach with large language models (LLMs) that can be used to solve diverse table tasks such as semantic parsing, question answering as well as classification problems. Traditionally, there existed separate models specialized for each task individually. It raises the question of how far can we go to build a unified model that works well on some table tasks without significant degradation on others. To that end, we attempt at creating a shared modeling approach in the pretraining stage with encoder-decoder style LLMs that can cater to diverse tasks. We evaluate our approach that continually pretrains and finetunes different model families of T5 with data from tables and surrounding context, on these downstream tasks at different model scales. Through multiple ablation studies, we observe that our pretraining with self-supervised objectives can significantly boost the performance of the models on these tasks. As an example of one improvement, we observe that the instruction finetuned public models which come specialized on text question answering (QA) and have been trained on table data still have room for improvement when it comes to table specific QA. Our work is the first attempt at studying the advantages of a unified approach to table specific pretraining when scaled from 770M to 11B sequence to sequence models while also comparing the instruction finetuned variants of the models.