Hybrid Querying Over Relational Databases and Large Language Models

TL;DR

This paper introduces SWAN, a benchmark for hybrid querying that combines relational databases with large language models, demonstrating promising results with GPT-4 Turbo in answering complex beyond-database questions.

Contribution

It presents the first cross-domain benchmark SWAN and explores two novel solutions for integrating LLMs with databases for complex querying.

Findings

Up to 40.0% execution accuracy with GPT-4 Turbo

48.2% data factuality achieved

Benchmark enables evaluation of hybrid querying approaches

Abstract

Database queries traditionally operate under the closed-world assumption, providing no answers to questions that require information beyond the data stored in the database. Hybrid querying using SQL offers an alternative by integrating relational databases with large language models (LLMs) to answer beyond-database questions. In this paper, we present the first cross-domain benchmark, SWAN, containing 120 beyond-database questions over four real-world databases. To leverage state-of-the-art language models in addressing these complex questions in SWAN, we present two solutions: one based on schema expansion and the other based on user defined functions. We also discuss optimization opportunities and potential future directions. Our evaluation demonstrates that using GPT-4 Turbo with few-shot prompts, one can achieves up to 40.0\% in execution accuracy and 48.2\% in data factuality. These results highlights both the potential and challenges for hybrid querying. We believe that our work will inspire further research in creating more efficient and accurate data systems that seamlessly integrate relational databases and large language models to address beyond-database questions.