Knowledge Distillation with Structured Chain-of-Thought for Text-to-SQL

TL;DR

This paper introduces Struct-SQL, a knowledge distillation framework that uses structured reasoning via query execution plans to improve small language models for Text-to-SQL tasks, achieving higher accuracy and fewer errors.

Contribution

It proposes a novel structured CoT approach for distilling large LLMs into smaller models using formal reasoning blueprints, enhancing reliability and performance.

Findings

Structured CoT improves SLM accuracy by 8.1% over unstructured methods.

Structured reasoning reduces syntactic errors in SQL generation.

Using formal blueprints enhances the reliability of small models in enterprise settings.

Abstract

Deploying accurate Text-to-SQL systems at the enterprise level faces a difficult trilemma involving cost, security and performance. Current solutions force enterprises to choose between expensive, proprietary Large Language Models (LLMs) and low-performing Small Language Models (SLMs). Efforts to improve SLMs often rely on distilling reasoning from large LLMs using unstructured Chain-of-Thought (CoT) traces, a process that remains inherently ambiguous. Instead, we hypothesize that a formal, structured reasoning representation provides a clearer, more reliable teaching signal, as the Text-to-SQL task requires explicit and precise logical steps. To evaluate this hypothesis, we propose Struct-SQL, a novel Knowledge Distillation (KD) framework that trains an SLM to emulate a powerful large LLM. Consequently, we adopt a query execution plan as a formal blueprint to derive this structured reasoning. Our SLM, distilled with structured CoT, achieves an absolute improvement of 8.1% over an unstructured CoT distillation baseline. A detailed error analysis reveals that a key factor in this gain is a marked reduction in syntactic errors. This demonstrates that teaching a model to reason using a structured logical blueprint is beneficial for reliable SQL generation in SLMs.