$R^3$: "This is My SQL, Are You With Me?" A Consensus-Based Multi-Agent System for Text-to-SQL Tasks

TL;DR

The paper introduces $R^3$, a consensus-based multi-agent system that significantly improves Text-to-SQL performance by leveraging multiple LLMs and a review-rebuttal-revision process, surpassing existing methods.

Contribution

It presents a novel multi-agent framework for Text-to-SQL that outperforms single LLM and other multi-agent systems, including chain-of-thought prompting, on benchmark datasets.

Findings

$R^3$ achieves 1.3% to 8.1% higher accuracy on Spider and Bird datasets.

$R^3$ outperforms chain-of-thought prompting by over 20% for Llama-3-8B.

$R^3$ surpasses GPT-3.5 performance on Spider development set.

Abstract

Large Language Models (LLMs) have demonstrated strong performance on various tasks. To unleash their power on the Text-to-SQL task, we propose $R^3$ (Review-Rebuttal-Revision), a consensus-based multi-agent system for Text-to-SQL tasks. $R^3$ outperforms the existing single LLM Text-to-SQL systems as well as the multi-agent Text-to-SQL systems by $1.3\%$ to $8.1\%$ on Spider and Bird. Surprisingly, we find that for Llama-3-8B, $R^3$ outperforms chain-of-thought prompting by over 20\%, even outperforming GPT-3.5 on the development set of Spider.