A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability

TL;DR

This paper thoroughly evaluates ChatGPT's zero-shot Text-to-SQL capabilities across multiple datasets, showing strong performance and potential for practical use despite some gaps with state-of-the-art models.

Contribution

It provides the first comprehensive analysis of ChatGPT's zero-shot Text-to-SQL performance on diverse benchmarks, highlighting its strengths and limitations.

Findings

ChatGPT performs well in zero-shot Text-to-SQL tasks.

In the ADVETA (RPL) scenario, ChatGPT outperforms fine-tuned SOTA models by 4.1%.

The data generated by ChatGPT is publicly available for further research.

Abstract

This paper presents the first comprehensive analysis of ChatGPT's Text-to-SQL ability. Given the recent emergence of large-scale conversational language model ChatGPT and its impressive capabilities in both conversational abilities and code generation, we sought to evaluate its Text-to-SQL performance. We conducted experiments on 12 benchmark datasets with different languages, settings, or scenarios, and the results demonstrate that ChatGPT has strong text-to-SQL abilities. Although there is still a gap from the current state-of-the-art (SOTA) model performance, considering that the experiment was conducted in a zero-shot scenario, ChatGPT's performance is still impressive. Notably, in the ADVETA (RPL) scenario, the zero-shot ChatGPT even outperforms the SOTA model that requires fine-tuning on the Spider dataset by 4.1\%, demonstrating its potential for use in practical applications. To support further research in related fields, we have made the data generated by ChatGPT publicly available at https://github.com/THU-BPM/chatgpt-sql.