DB-GPT-Hub: Towards Open Benchmarking Text-to-SQL Empowered by Large Language Models

TL;DR

DB-GPT-Hub introduces an open benchmark suite for fine-tuning large language models on text-to-SQL tasks, enabling systematic evaluation and comparison of tuning versus prompting methods in this domain.

Contribution

It provides a standardized, extensible benchmark for fine-tuning LLMs on text-to-SQL, addressing the high computational cost barrier and facilitating research in this area.

Findings

Fine-tuning LLMs can outperform prompting approaches in text-to-SQL.

Benchmark reveals performance boundaries of tuning methods.

Open-source code supports easy extension and experimentation.

Abstract

Large language models (LLMs) becomes the dominant paradigm for the challenging task of text-to-SQL. LLM-empowered text-to-SQL methods are typically categorized into prompting-based and tuning approaches. Compared to prompting-based methods, benchmarking fine-tuned LLMs for text-to-SQL is important yet under-explored, partially attributed to the prohibitively high computational cost. In this paper, we present DB-GPT-Hub, an open benchmark suite for LLM-empowered text-to-SQL, which primarily focuses on tuning LLMs at large scales. The proposed benchmark consists of: 1. a standardized and comprehensive evaluation of text-to-SQL tasks by fine-tuning medium to large-sized open LLMs; 2. a modularized and easy-to-extend codebase with mainstream LLMs and experimental scenarios supported, which prioritizes fine-tuning methods but can be easily extended to prompt-based setting. Our work investigates the potential gains and the performance boundaries of tuning approaches, compared to prompting approaches and explores optimal solutions tailored to specific scenarios. We hope DB-GPT-Hub, along with these findings, enables further research and broad applications that would otherwise be difficult owing to the absence of a dedicated open benchmark. The project code has been released at https://github.com/eosphoros-ai/DB-GPT-Hub.