Optimizing Small Language Models for NL2SQL via Chain-of-Thought Fine-Tuning

TL;DR

This paper shows that fine-tuning small language models with Chain-of-Thought reasoning significantly improves NL2SQL accuracy, offering a cost-effective alternative to large models for enterprise data access.

Contribution

It demonstrates that small models can achieve near-production NL2SQL performance through targeted fine-tuning and reasoning pattern transfer, reducing costs and inference latency.

Findings

Fine-tuning small models improves accuracy from 36% to 45%.

Enriching datasets with Chain-of-Thought reasoning boosts accuracy to 54.5%.

Large models show negligible gains from fine-tuning on standard datasets.

Abstract

Translating Natural Language to SQL (NL2SQL) remains a critical bottleneck for democratization of data in enterprises. Although Large Language Models (LLMs) like Gemini 2.5 and other LLMs have demonstrated impressive zero-shot capabilities, their high inference costs limit deployment at scale. This paper explores the efficacy of fine-tuning both large and small language models on NL2SQL tasks. Our research reveals a counter-intuitive scaling phenomenon. Fine-tuning large models (Gemini 2.5 Flash/Lite) on standard datasets yields negligible returns, often leading to overfitting on complex queries. Conversely, small models (Qwen) show significant gains. Fine-tuning improved the small model baseline from 36% to 45%, and further enriching the dataset with explicit Chain-of-Thought (CoT) reasoning surged accuracy to 54.5%(Fig 2). While this is still lower than the accuracy of large models like Gemini 2.5 , it does serve the business goal of significant cost reduction, latency in inference time and also meeting the business critical performance accuracy threshold.This paper demonstrates that transferring reasoning patterns enables compute-efficient smaller models to approach production-grade performance.