An Agentic System for Schema Aware NL2SQL Generation

TL;DR

This paper introduces a schema-aware agentic NL2SQL system that uses small language models with a fallback to large models, significantly reducing costs while maintaining high accuracy in translating natural language to SQL.

Contribution

The proposed system strategically combines small and large language models to improve efficiency and reduce costs in NL2SQL tasks, addressing deployability concerns.

Findings

Achieves 47.78% execution accuracy on BIRD benchmark.

Reduces operational costs by over 90% compared to LLM-only systems.

Resolves approximately 67% of queries using local SLMs.

Abstract

The natural language to SQL (NL2SQL) task plays a pivotal role in democratizing data access by enabling non-expert users to interact with relational databases through intuitive language. While recent frameworks have enhanced translation accuracy via task specialization, their reliance on Large Language Models (LLMs) raises significant concerns regarding computational overhead, data privacy, and real-world deployability in resource-constrained environments. To address these challenges, we propose a schema based agentic system that strategically employs Small Language Models (SLMs) as primary agents, complemented by a selective LLM fallback mechanism. The LLM is invoked only upon detection of errors in SLM-generated output, the proposed system significantly minimizes computational expenditure. Experimental results on the BIRD benchmark demonstrate that our system achieves an execution accuracy of 47.78% and a validation efficiency score of 51.05%, achieving over 90% cost reduction compared to LLM-centric baselines as approximately 67% of queries are resolved using local SLMs. The system achieves an average cost per query of 0.0085 compared to 0.094 for LLM-only systems, achieving near-zero operational costs for locally executed queries. [Github repository: https://github.com/mindslab25/CESMA.]