Agent-Agnostic Evaluation of SQL Accuracy in Production Text-to-SQL Systems

TL;DR

This paper introduces STEF, a schema-agnostic evaluation framework for Text-to-SQL systems that enables production-level monitoring and improvement without relying on database schemas or reference queries.

Contribution

The paper presents STEF, a novel production-native evaluation system for Text-to-SQL that operates solely on natural language inputs and generated SQL, removing schema dependency.

Findings

STEF provides a 0-100 accuracy score based on semantic alignment.

Enables continuous monitoring and feedback for production Text-to-SQL agents.

Handles schema variations and default heuristics for robust evaluation.

Abstract

Text-to-SQL (T2SQL) evaluation in production environments poses fundamental challenges that existing benchmarks do not address. Current evaluation methodologies whether rule-based SQL matching or schema-dependent semantic parsers assume access to ground-truth queries and structured database schema, constraints that are rarely satisfied in real-world deployments. This disconnect leaves production T2SQL agents largely unevaluated beyond developer-time testing, creating silent quality degradation with no feedback mechanism for continuous improvement. We present STEF (Schema-agnostic Text-to-SQL Evaluation Framework), a production-native evaluation system that operates exclusively on natural language inputs the user question, an enriched reformulation, and the generated SQL without requiring database schema or reference queries. STEF extracts semantic specifications from both natural language and SQL representations, performs normalized feature alignment, and produces an interpretable 0 to 100 accuracy score via a composite metric that encompasses filter alignment, semantic verdict, and confidence of the evaluator. Key contributions include: enriched question quality validation as a first-class evaluation signal, configurable application-specific rule injection via prompt templating, and production-robust normalization handling GROUP BY tolerance, ORDER BY defaults, and LIMIT heuristics. Empirical results demonstrate that STEF enables continuous production monitoring and agent improvement feedback loops without schema dependency, making structured query evaluation viable at scale for the first time.