Exploring Chain-of-Thought Style Prompting for Text-to-SQL

TL;DR

This paper investigates how chain-of-thought prompting can improve large language models' reasoning for text-to-SQL tasks, proposing a new method that significantly enhances performance over existing prompting techniques.

Contribution

The paper introduces a novel chain-of-thought prompting approach tailored for text-to-SQL, demonstrating substantial performance improvements over standard and least-to-most prompting methods.

Findings

Iterative prompting may be unnecessary for text-to-SQL.

Detailed reasoning steps can cause error propagation.

Proposed method improves Spider dataset scores by over 5 points.

Abstract

In-context learning with large language models (LLMs) has recently caught increasing attention due to its superior few-shot performance on various tasks. However, its performance on text-to-SQL parsing still has much room for improvement. In this paper, we hypothesize that a crucial aspect of LLMs to improve for text-to-SQL parsing is their multi-step reasoning ability. Thus, we systematically study how to enhance LLMs' reasoning ability through chain of thought (CoT) style prompting, including the original chain-of-thought prompting (Wei et al., 2022b) and least-to-most prompting (Zhou et al., 2023). Our experiments demonstrate that iterative prompting as in Zhou et al. (2023) may be unnecessary for text-to-SQL parsing, and using detailed reasoning steps tends to have more error propagation issues. Based on these findings, we propose a new CoT-style prompting method for text-to-SQL parsing. It brings 5.2 and 6.5 point absolute gains on the Spider development set and the Spider Realistic set, respectively, compared to the standard prompting method without reasoning steps; 2.4 and 1.5 point absolute gains, compared to the least-to-most prompting method.