SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation

TL;DR

This paper introduces SQLStructEval, a framework for analyzing the structural reliability of LLM-generated SQL queries, revealing variability issues and proposing a pipeline to improve consistency and accuracy.

Contribution

The work presents a novel framework for structural analysis of LLM-generated SQL and demonstrates how structured generation improves reliability.

Findings

LLMs often produce structurally diverse SQL queries for the same input.

Surface-level input changes trigger structural variance in generated SQL.

Structured query generation improves execution accuracy and structural consistency.

Abstract

Despite strong performance on Text-to-SQL benchmarks, it remains unclear whether LLM-generated SQL programs are structurally reliable. In this work, we investigate the structural behavior of LLM-generated SQL queries and introduce SQLStructEval, a framework for analyzing program structures through canonical abstract syntax tree (AST) representations. Our experiments on the Spider benchmark show that modern LLMs often produce structurally diverse queries for the same input, even when execution results are correct, and that such variance is frequently triggered by surface-level input changes such as paraphrases or schema presentation. We further show that generating queries in a structured space via a compile-style pipeline can improve both execution accuracy and structural consistency. These findings suggest that structural reliability is a critical yet overlooked dimension for evaluating LLM-based program generation systems. Our code is available at https://anonymous.4open.science/r/StructEval-2435.