SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended)

TL;DR

This paper presents SQL-PaLM, an advanced framework leveraging large language models with few-shot prompting and instruction fine-tuning to significantly improve Text-to-SQL translation performance on benchmark datasets.

Contribution

The paper introduces SQL-PaLM, a comprehensive approach combining data augmentation, multi-paradigm ensemble, and efficient database navigation to enhance Text-to-SQL accuracy with LLMs.

Findings

Substantial performance improvements on Spider and BIRD benchmarks.

Effective use of execution-based error filtering and test-time selection.

Enhanced handling of complex databases with many tables and columns.

Abstract

Text-to-SQL, the process of translating natural language into Structured Query Language (SQL), represents a transformative application of large language models (LLMs), potentially revolutionizing how humans interact with data. This paper introduces the SQL-PaLM framework, a comprehensive solution for understanding and enhancing Text-to-SQL using LLMs, using in the learning regimes of few-shot prompting and instruction fine-tuning. With few-shot prompting, we explore the effectiveness of consistency decoding with execution-based error filtering. With instruction fine-tuning, we delve deep in understanding the critical paradigms that influence the performance of tuned LLMs. In particular, we investigate how performance can be improved through expanded training data coverage and diversity, synthetic data augmentation, and integrating query-specific database content. We propose a test-time selection method to further refine accuracy by integrating SQL outputs from multiple paradigms with execution feedback as guidance. Additionally, we tackle the practical challenge of navigating intricate databases with a significant number of tables and columns, proposing efficient techniques for accurately selecting relevant database elements to enhance Text-to-SQL performance. Our holistic approach yields substantial advancements in Text-to-SQL, as demonstrated on two key public benchmarks, Spider and BIRD. Through comprehensive ablations and error analyses, we shed light on the strengths and weaknesses of our framework, offering valuable insights into Text-to-SQL's future work.