Lucy: Think and Reason to Solve Text-to-SQL

TL;DR

This paper introduces Lucy, a framework that enhances large language models with automated reasoning to improve zero-shot text-to-SQL performance on complex enterprise databases.

Contribution

The paper presents a novel framework combining LLMs with automated reasoning to better handle complex database schemas in text-to-SQL tasks.

Findings

Outperforms state-of-the-art zero-shot methods on complex benchmarks

Effectively handles large enterprise database schemas

Improves reasoning about complex database constraints

Abstract

Large Language Models (LLMs) have made significant progress in assisting users to query databases in natural language. While LLM-based techniques provide state-of-the-art results on many standard benchmarks, their performance significantly drops when applied to large enterprise databases. The reason is that these databases have a large number of tables with complex relationships that are challenging for LLMs to reason about. We analyze challenges that LLMs face in these settings and propose a new solution that combines the power of LLMs in understanding questions with automated reasoning techniques to handle complex database constraints. Based on these ideas, we have developed a new framework that outperforms state-of-the-art techniques in zero-shot text-to-SQL on complex benchmarks