Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQL

TL;DR

This paper introduces KID, a knowledge distillation method that improves the efficiency and performance of autoregressive language models for text-to-SQL tasks by using imperfect data to simulate inference effects.

Contribution

The paper proposes KID, a novel knowledge distillation approach that effectively leverages imperfect data to enhance model performance and training efficiency in text-to-SQL tasks.

Findings

KID achieves up to +5.83% average score improvement.

KID enhances training efficiency while maintaining performance.

Extensive experiments on 5 benchmarks validate KID's effectiveness.

Abstract

Large Language Models (LLMs) have shown promising performance in text-to-SQL, which involves translating natural language questions into SQL queries. However, current text-to-SQL LLMs are computationally expensive and challenging to deploy in real-world applications, highlighting the importance of compressing them. To achieve this goal, knowledge distillation (KD) is a common approach, which aims to distill the larger teacher model into a smaller student model. While numerous KD methods for autoregressive LLMs have emerged recently, it is still under-explored whether they work well in complex text-to-SQL scenarios. To this end, we conduct a series of analyses and reveal that these KD methods generally fall short in balancing performance and efficiency. In response to this problem, we propose to improve the KD with Imperfect Data, namely KID, which effectively boosts the performance without introducing much training budget. The core of KID is to efficiently mitigate the training-inference mismatch by simulating the cascading effect of inference in the imperfect training data. Extensive experiments on 5 text-to-SQL benchmarks show that, KID can not only achieve consistent and significant performance gains (up to +5.83% average score) across all model types and sizes, but also effectively improve the training efficiency.