Structure-Grounded Pretraining for Text-to-SQL

TL;DR

This paper introduces StruG, a weakly supervised pretraining framework for text-to-SQL that improves text-table alignment understanding, especially in realistic scenarios with less explicit column references.

Contribution

StruG is a novel pretraining method that leverages a parallel text-table corpus with new prediction tasks to enhance text-to-SQL models' alignment capabilities.

Findings

StruG significantly outperforms BERT-LARGE across all evaluation settings.

Achieves comparable results to GRAPPA on Spider dataset.

Excels on more realistic datasets with less explicit column mentions.

Abstract

Learning to capture text-table alignment is essential for tasks like text-to-SQL. A model needs to correctly recognize natural language references to columns and values and to ground them in the given database schema. In this paper, we present a novel weakly supervised Structure-Grounded pretraining framework (StruG) for text-to-SQL that can effectively learn to capture text-table alignment based on a parallel text-table corpus. We identify a set of novel prediction tasks: column grounding, value grounding and column-value mapping, and leverage them to pretrain a text-table encoder. Additionally, to evaluate different methods under more realistic text-table alignment settings, we create a new evaluation set Spider-Realistic based on Spider dev set with explicit mentions of column names removed, and adopt eight existing text-to-SQL datasets for cross-database evaluation. STRUG brings significant improvement over BERT-LARGE in all settings. Compared with existing pretraining methods such as GRAPPA, STRUG achieves similar performance on Spider, and outperforms all baselines on more realistic sets. The Spider-Realistic dataset is available at https://doi.org/10.5281/zenodo.5205322.