FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark

TL;DR

FLEX is a new evaluation metric for text-to-SQL systems that uses large language models to emulate expert human judgment, significantly reducing false positives and negatives in performance assessment.

Contribution

This paper introduces FLEX, an expert-level evaluation metric for text-to-SQL that improves agreement with human judgment and addresses limitations of existing metrics.

Findings

Performance scores increase by over 2.6 points on average.

Annotation quality issues cause underestimation of model performance.

Model performance on difficult questions is often overestimated.

Abstract

Text-to-SQL systems have become crucial for translating natural language into SQL queries in various industries, enabling non-technical users to perform complex data operations. The need for accurate evaluation methods has increased as these systems have grown more sophisticated. However, the Execution Accuracy (EX), the most prevalent evaluation metric, still shows many false positives and negatives. Thus, this paper introduces FLEX (False-Less EXecution), a novel approach to evaluating text-to-SQL systems using large language models (LLMs) to emulate human expert-level evaluation of SQL queries. Our metric improves agreement with human experts (from 62 to 87.04 in Cohen's kappa) with comprehensive context and sophisticated criteria. Our extensive experiments yield several key insights: (1) Models' performance increases by over 2.6 points on average, substantially affecting rankings on Spider and BIRD benchmarks; (2) The underestimation of models in EX primarily stems from annotation quality issues; and (3) Model performance on particularly challenging questions tends to be overestimated. This work contributes to a more accurate and nuanced evaluation of text-to-SQL systems, potentially reshaping our understanding of state-of-the-art performance in this field.