SLM-SQL: An Exploration of Small Language Models for Text-to-SQL

TL;DR

This paper investigates the potential of small language models for Text-to-SQL tasks by applying post-training techniques and a corrective self-consistency inference method, achieving significant performance improvements.

Contribution

The study introduces a novel approach combining supervised fine-tuning and reinforcement learning-based post-training for small language models in Text-to-SQL tasks, demonstrating their effectiveness.

Findings

0.5B model achieved 56.87% execution accuracy.

1.5B model achieved 67.08% execution accuracy.

Significant performance gains with small models using proposed methods.

Abstract

Large language models (LLMs) have demonstrated strong performance in translating natural language questions into SQL queries (Text-to-SQL). In contrast, small language models (SLMs) ranging from 0.5B to 1.5B parameters currently underperform on Text-to-SQL tasks due to their limited logical reasoning capabilities. However, SLMs offer inherent advantages in inference speed and suitability for edge deployment. To explore their potential in Text-to-SQL applications, we leverage recent advancements in post-training techniques. Specifically, we used the open-source SynSQL-2.5M dataset to construct two derived datasets: SynSQL-Think-916K for SQL generation and SynSQL-Merge-Think-310K for SQL merge revision. We then applied supervised fine-tuning and reinforcement learning-based post-training to the SLM, followed by inference using a corrective self-consistency approach. Experimental results validate the effectiveness and generalizability of our method, SLM-SQL. On the BIRD development set, the five evaluated models achieved an average improvement of 31.4 points. Notably, the 0.5B model reached 56.87\% execution accuracy (EX), while the 1.5B model achieved 67.08\% EX. We will release our dataset, model, and code to github: https://github.com/CycloneBoy/slm_sql.