FloodSQL-Bench: A Retrieval-Augmented Benchmark for Geospatially-Grounded Text-to-SQL

TL;DR

FloodSQL-Bench is a new benchmark designed to evaluate large language models' ability to generate accurate, geospatially-grounded SQL queries in flood management, reflecting real-world complexity and multi-table reasoning.

Contribution

It introduces a domain-specific, multi-table, geospatially-grounded benchmark for Text-to-SQL, addressing limitations of existing benchmarks and facilitating research in high-stakes disaster management applications.

Findings

Large language models show varying performance across difficulty tiers.

The benchmark reveals challenges in multi-table and geospatial reasoning.

FLOODSQL-BENCH provides a realistic testbed for future research.

Abstract

Existing Text-to-SQL benchmarks primarily focus on single-table queries or limited joins in general-purpose domains, and thus fail to reflect the complexity of domain-specific, multi-table and geospatial reasoning, To address this limitation, we introduce FLOODSQL-BENCH, a geospatially grounded benchmark for the flood management domain that integrates heterogeneous datasets through key-based, spatial, and hybrid joins. The benchmark captures realistic flood-related information needs by combining social, infrastructural, and hazard data layers. We systematically evaluate recent large language models with the same retrieval-augmented generation settings and measure their performance across difficulty tiers. By providing a unified, open benchmark grounded in real-world disaster management data, FLOODSQL-BENCH establishes a practical testbed for advancing Text-to-SQL research in high-stakes application domains.