SQL-Commenter: Aligning Large Language Models for SQL Comment Generation with Direct Preference Optimization

TL;DR

This paper introduces SQL-Commenter, a method that improves SQL comment generation by combining extensive dataset construction, continual pre-training, supervised fine-tuning, and preference optimization, significantly outperforming existing models.

Contribution

It presents a novel approach integrating preference optimization with large language models for more accurate and context-aware SQL comment generation.

Findings

SQL-Commenter outperforms baselines on BLEU-4, METEOR, and ROUGE-L metrics.

Human evaluation confirms higher quality comments in correctness, completeness, and naturalness.

Constructed a comprehensive dataset of complex SQL queries with expert-verified comments.

Abstract

SQL query comprehension is a significant challenge due to complex syntax, diverse join types, and deep nesting. Many queries lack adequate comments, severely hindering code readability, maintainability, and knowledge transfer. Automated SQL comment generation faces two main challenges: limited datasets that inadequately represent complex real-world queries, and Large Language Models' (LLMs) insufficient understanding of SQL-specific semantics. Our empirical analysis shows that even after continual pre-training and supervised fine-tuning, LLMs struggle with complex SQL semantics, yielding inaccurate comments. To address this, we propose SQL-Commenter, an advanced method based on LLaMA-3.1-8B. We first construct a comprehensive dataset of complex SQL queries with expert-verified comments. Next, we perform continual pre-training on a large SQL corpus to enhance the LLM's syntax and semantic understanding, followed by supervised fine-tuning. Finally, we introduce Direct Preference Optimization (DPO) using human feedback. SQL-Commenter utilizes a preference-based loss function to favor preferred outputs, enhancing fine-grained semantic learning and context-dependent quality assessment. Evaluated on the Spider and Bird benchmarks, SQL-Commenter significantly outperforms state-of-the-art baselines. On average, it surpasses the strongest baseline (Qwen3-14B) by 9.29, 4.99, and 13.23 percentage points on BLEU-4, METEOR, and ROUGE-L, respectively. Moreover, human evaluation demonstrates the superior quality of comments generated by SQL-Commenter in terms of correctness, completeness, and naturalness.