Beyond Text-to-SQL: Can LLMs Really Debug Enterprise ETL SQL?

TL;DR

This paper introduces OurBench, a comprehensive benchmark for evaluating large language models' ability to debug complex enterprise SQL code, highlighting significant performance gaps and challenges.

Contribution

It presents a novel automated workflow for creating realistic SQL debugging benchmarks and an evaluation framework tailored for enterprise SQL reasoning.

Findings

Best model achieves only ~36% accuracy on syntax errors.

Most models score below 20% on semantic errors.

The benchmark includes highly complex, large-scale SQL queries.

Abstract

SQL is central to enterprise data engineering, yet generating fully correct SQL code in a single attempt remains difficult, even for experienced developers and advanced text-to-SQL LLMs, often requiring multiple debugging iterations. We introduce OurBench, the first benchmark for enterprise-level SQL reasoning and debugging. Our benchmark is built on two key innovations: (1) an automated construction workflow that uses reverse engineering to systematically inject realistic bugs into large-scale SQL code, enabling scalable and diverse benchmark generation; and (2) an execution-free evaluation framework tailored to enterprise settings, providing fast, accurate, and resource-efficient assessment. OurBench comprises 469 OurBenchSyn queries featuring syntax errors with explicit error messages, and 516 OurBenchSem queries targeting semantic errors in which the code fails to meet user intent. The queries are highly complex, averaging over 140 lines and featuring deep and wide abstract syntax trees. Evaluation of nearly 30 LLMs reveals a substantial performance gap: the best-performing model, Claude-4-Sonnet, achieves only 36.46 percent accuracy on OurBenchSyn and 32.17 percent on OurBenchSem, while most models score below 20 percent. We further explore four solution strategies, identify key challenges, and outline promising directions for enterprise SQL debugging with LLMs.