A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization

TL;DR

This paper introduces SQLova, a novel NL2SQL model leveraging BERT with table-aware contextualization, achieving near-human performance on WikiSQL and outperforming previous methods significantly.

Contribution

It presents SQLova, the first model to combine BERT with table-aware contextualization for NL2SQL, surpassing prior state-of-the-art results and providing comprehensive analysis of dataset and model performance.

Findings

SQLova achieves 8.2% and 2.5% improvements in logical form and execution accuracy.

The model exceeds human performance by 1.3% in execution accuracy.

BERT with seq2seq decoder performs poorly, highlighting the importance of model design.

Abstract

We present SQLova, the first Natural-language-to-SQL (NL2SQL) model to achieve human performance in WikiSQL dataset. We revisit and discuss diverse popular methods in NL2SQL literature, take a full advantage of BERT {Devlin et al., 2018) through an effective table contextualization method, and coherently combine them, outperforming the previous state of the art by 8.2% and 2.5% in logical form and execution accuracy, respectively. We particularly note that BERT with a seq2seq decoder leads to a poor performance in the task, indicating the importance of a careful design when using such large pretrained models. We also provide a comprehensive analysis on the dataset and our model, which can be helpful for designing future NL2SQL datsets and models. We especially show that our model's performance is near the upper bound in WikiSQL, where we observe that a large portion of the evaluation errors are due to wrong annotations, and our model is already exceeding human performance by 1.3% in execution accuracy.