Falcon: A Comprehensive Chinese Text-to-SQL Benchmark for Enterprise-Grade Evaluation

TL;DR

Falcon introduces a comprehensive Chinese Text-to-SQL benchmark tailored for enterprise environments, emphasizing real-world complexities like schema linking and colloquial language, to evaluate and improve large-scale models.

Contribution

The paper presents Falcon, a novel Chinese Text-to-SQL benchmark with enterprise-specific features, detailed annotations, and evaluation tools, addressing challenges in schema linking and colloquial language understanding.

Findings

Current models achieve at most 50% accuracy on Falcon.

Major errors stem from schema linking and colloquial language mapping.

Falcon highlights the need for improved models in enterprise-specific Chinese SQL tasks.

Abstract

We introduce Falcon, a cross-domain Chinese text-to-SQL benchmark grounded in an enterprise-compatible dialect (MaxCompute/Hive). It contains 600 Chinese questions over 28 databases; 77% require multi-table reasoning and over half touch more than four tables. Each example is annotated along SQL-computation features and Chinese semantics. For evaluation, we release a robust execution comparator and an automated evaluation pipeline, under which all current state-of-the-art large-scale models (including Deepseek) achieve accuracies of at most 50%. Major errors originate from two sources: (1) schema linking in large enterprise landscapes - hundreds of tables, denormalized fields, ambiguous column names, implicit foreign-key relations and domain-specific synonyms that make correct join/column selection difficult; and (2) mapping concise, colloquial Chinese into the exact operators and predicates required for analytics - e.g., choosing the correct aggregation and group-by keys, expressing time windows and granularities, applying unit conversions, handling NULLs and data-quality rules, and formulating nested or windowed subqueries. Falcon therefore targets Chinese-specific semantics and enterprise dialects (abbreviations, business jargon, fuzzy entity references) and provides a reproducible middle ground before full production deployment by using realistic enterprise schemas, query templates, an execution comparator, and an automated evaluation pipeline for end-to-end validation.