CA-SQL: Complexity-Aware Inference Time Reasoning for Text-to-SQL via Exploration and Compute Budget Allocation

TL;DR

CA-SQL introduces a complexity-aware, exploration-driven approach for Text-to-SQL inference, dynamically allocating resources based on task difficulty to improve performance on challenging benchmarks.

Contribution

It proposes a novel pipeline that uses task difficulty estimation, evolutionary-inspired prompt seeding, and a voting mechanism to enhance LLM-based Text-to-SQL reasoning.

Findings

Achieves 51.72% score on challenging BIRD benchmark tasks.

Outperforms other in-context learning methods with smaller models.

Attains 61.06% execution accuracy and 68.77% Soft F1 score on BIRD dataset.

Abstract

While recent advancements in inference-time learning have improved LLM reasoning on Text-to-SQL tasks, current solutions still struggle to perform well on the most challenging tasks in the Bird-Bench (BIRD) benchmark. This is due to inadequate solution space exploration, which is necessary to uncover promising candidate queries that can be further refined to produce the correct output. To address this challenge, we introduce CA-SQL, a novel Text-to-SQL pipeline that utilizes the estimated difficulty of a task to dynamically scale the breadth of the exploration for generating solution candidates. In addition, we use a custom prompt seeding method, based on principles of evolutionary search, to further elicit exploratory behavior from the base LLM and a novel voting method to select the best candidate solution at the end of the search. Experiments demonstrate that our solution achieves a state-of-the-art score of 51.72% on the "challenging" tier of BIRD development set problems, using only GPT-4o-mini, out-performing other in-context learning approaches, even those that leverage larger models. Overall, our method attains a competitive 61.06% execution accuracy and 68.77% Soft F1 score on the BIRD development dataset.