Using LLM to select the right SQL Query from candidates

TL;DR

This paper introduces a novel re-ranking method for Text-to-SQL models that uses automatically generated test cases to select the most accurate SQL query from candidate lists, improving model performance.

Contribution

It proposes an automatic test case generation approach for Text-to-SQL re-ranking, leveraging LLMs to generate databases and predict execution results, which enhances candidate selection accuracy.

Findings

Achieved a 3.6% performance improvement on the Spider dataset.

Demonstrated effectiveness of test case-based re-ranking for state-of-the-art models.

Showed that easy-to-understand prompts improve test case generation.

Abstract

Text-to-SQL models can generate a list of candidate SQL queries, and the best query is often in the candidate list, but not at the top of the list. An effective re-rank method can select the right SQL query from the candidate list and improve the model's performance. Previous studies on code generation automatically generate test cases and use them to re-rank candidate codes. However, automatic test case generation for text-to-SQL is an understudied field. We propose an automatic test case generation method that first generates a database and then uses LLMs to predict the ground truth, which is the expected execution results of the ground truth SQL query on this database. To reduce the difficulty for LLMs to predict, we conduct experiments to search for ways to generate easy databases for LLMs and design easy-to-understand prompts. Based on our test case generation method, we propose a re-rank method to select the right SQL query from the candidate list. Given a candidate list, our method can generate test cases and re-rank the candidate list according to their pass numbers on these test cases and their generation probabilities. The experiment results on the validation dataset of Spider show that the performance of some state-of-the-art models can get a 3.6\% improvement after applying our re-rank method.