SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps

TL;DR

SQLSpace is a novel, interpretable representation for text-to-SQL tasks that enhances evaluation, understanding, and performance improvement by revealing detailed insights beyond traditional metrics.

Contribution

The paper introduces SQLSpace, a compact, human-interpretable representation for text-to-SQL examples, enabling detailed analysis and targeted model improvements.

Findings

Reveals compositional differences between benchmarks.

Identifies granular performance patterns.

Supports targeted query rewriting for better accuracy.

Abstract

We introduce SQLSpace, a human-interpretable, generalizable, compact representation for text-to-SQL examples derived with minimal human intervention. We demonstrate the utility of these representations in evaluation with three use cases: (i) closely comparing and contrasting the composition of popular text-to-SQL benchmarks to identify unique dimensions of examples they evaluate, (ii) understanding model performance at a granular level beyond overall accuracy scores, and (iii) improving model performance through targeted query rewriting based on learned correctness estimation. We show that SQLSpace enables analysis that would be difficult with raw examples alone: it reveals compositional differences between benchmarks, exposes performance patterns obscured by accuracy alone, and supports modeling of query success.