TURL: Table Understanding through Representation Learning

TL;DR

TURL introduces a pre-training and fine-tuning framework for relational Web tables, leveraging a structure-aware Transformer and a novel Masked Entity Recovery objective to improve performance across multiple table understanding tasks.

Contribution

The paper presents a universal pre-trained model for table understanding that requires minimal task-specific adjustments, advancing beyond heavily-engineered prior methods.

Findings

Outperforms existing methods on 6 table understanding tasks

Generalizes well across diverse table-related tasks

Uses a novel Masked Entity Recovery pre-training objective

Abstract

Relational tables on the Web store a vast amount of knowledge. Owing to the wealth of such tables, there has been tremendous progress on a variety of tasks in the area of table understanding. However, existing work generally relies on heavily-engineered task-specific features and model architectures. In this paper, we present TURL, a novel framework that introduces the pre-training/fine-tuning paradigm to relational Web tables. During pre-training, our framework learns deep contextualized representations on relational tables in an unsupervised manner. Its universal model design with pre-trained representations can be applied to a wide range of tasks with minimal task-specific fine-tuning. Specifically, we propose a structure-aware Transformer encoder to model the row-column structure of relational tables, and present a new Masked Entity Recovery (MER) objective for pre-training to capture the semantics and knowledge in large-scale unlabeled data. We systematically evaluate TURL with a benchmark consisting of 6 different tasks for table understanding (e.g., relation extraction, cell filling). We show that TURL generalizes well to all tasks and substantially outperforms existing methods in almost all instances.