Bridge the Gap between Language models and Tabular Understanding

TL;DR

This paper introduces UTP, a pre-training approach for tabular language models that supports multiple input types and uses contrastive learning to improve table-text understanding and task performance.

Contribution

UTP is a novel pre-training method that dynamically handles table, text, and combined inputs, bridging the gap between pre-training and fine-tuning phases.

Findings

UTP outperforms existing models on table retrieval tasks.

UTP achieves superior results on table question answering.

The approach improves alignment between table and text modalities.

Abstract

Table pretrain-then-finetune paradigm has been proposed and employed at a rapid pace after the success of pre-training in the natural language domain. Despite the promising findings in tabular pre-trained language models (TPLMs), there is an input gap between pre-training and fine-tuning phases. For instance, TPLMs jointly pre-trained with table and text input could be effective for tasks also with table-text joint input like table question answering, but it may fail for tasks with only tables or text as input such as table retrieval. To this end, we propose UTP, an approach that dynamically supports three types of multi-modal inputs: table-text, table, and text. Specifically, UTP is pre-trained with two strategies: (1) We first utilize a universal mask language modeling objective on each kind of input, enforcing the model to adapt various inputs. (2) We then present Cross-Modal Contrastive Regularization (CMCR), which utilizes contrastive learning to encourage the consistency between table-text cross-modality representations via unsupervised instance-wise training signals during pre-training. By these means, the resulting model not only bridges the input gap between pre-training and fine-tuning but also advances in the alignment of table and text. Extensive results show UTP achieves superior results on uni-modal input tasks (e.g., table retrieval) and cross-modal input tasks (e.g., table question answering).