Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrieval

TL;DR

This paper introduces a novel LLM-based framework for Text-to-SQL that uses de-semanticization and skeleton retrieval to improve example selection, enhancing performance and generalization across multiple benchmarks.

Contribution

It proposes a de-semanticization mechanism and schema filtering approach to better retrieve demonstration examples, significantly improving Text-to-SQL performance.

Findings

Outperforms state-of-the-art models on three benchmarks.

Demonstrates strong generalization across domains.

Effective retrieval of structurally similar examples enhances accuracy.

Abstract

Text-to-SQL is a task that converts a natural language question into a structured query language (SQL) to retrieve information from a database. Large language models (LLMs) work well in natural language generation tasks, but they are not specifically pre-trained to understand the syntax and semantics of SQL commands. In this paper, we propose an LLM-based framework for Text-to-SQL which retrieves helpful demonstration examples to prompt LLMs. However, questions with different database schemes can vary widely, even if the intentions behind them are similar and the corresponding SQL queries exhibit similarities. Consequently, it becomes crucial to identify the appropriate SQL demonstrations that align with our requirements. We design a de-semanticization mechanism that extracts question skeletons, allowing us to retrieve similar examples based on their structural similarity. We also model the relationships between question tokens and database schema items (i.e., tables and columns) to filter out scheme-related information. Our framework adapts the range of the database schema in prompts to balance length and valuable information. A fallback mechanism allows for a more detailed schema to be provided if the generated SQL query fails. Ours outperforms state-of-the-art models and demonstrates strong generalization ability on three cross-domain Text-to-SQL benchmarks.