ConstrainedSQL: Training LLMs for Text2SQL via Constrained Reinforcement Learning

TL;DR

This paper introduces a constrained reinforcement learning framework for training Text2SQL language models, emphasizing natural reward signals and theoretical guarantees, leading to improved performance over existing RL methods.

Contribution

It proposes a novel constrained RL approach with interpretable rewards and constraints, providing theoretical guarantees and demonstrating superior results on Text2SQL datasets.

Findings

Improved Text2SQL performance over state-of-the-art RL methods.

Theoretical guarantees for the constrained RL framework.

Effective balancing of reward and constraint signals during training.

Abstract

Reinforcement learning (RL) has demonstrated significant promise in enhancing the reasoning capabilities of Text2SQL LLMs, especially with advanced algorithms such as GRPO and DAPO. However, the performance of these methods is highly sensitive to the design of reward functions. Inappropriate rewards can lead to reward hacking, where models exploit loopholes in the reward structure to achieve high scores without genuinely solving the task. This work considers a constrained RL framework for Text2SQL that incorporates natural and interpretable reward and constraint signals, while dynamically balancing trade-offs among them during the training. We establish the theoretical guarantees of our constrained RL framework and our numerical experiments on the well-known Text2SQL datasets substantiate the improvement of our approach over the state-of-the-art RL-trained LLMs.