CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics

TL;DR

CricBench is a novel multilingual benchmark suite designed to evaluate the SQL generation capabilities of large language models in cricket analytics across various formats and languages.

Contribution

It introduces the first Text-to-SQL benchmark for cricket, with a curated dataset in four languages and evaluation of seven diverse models on domain-specific tasks.

Findings

No single model dominates across all cricket formats.

Models show high syntactic but low semantic accuracy in SQL generation.

Significant domain gap exists compared to existing benchmarks.

Abstract

Cricket is the second most popular sport worldwide, with billions of fans seeking advanced statistical insights unavailable through standard web searches. Although LLMs have advanced significantly in Text-to-SQL tasks, their capability to handle domain-specific nuances and multilingual requirements in sports analytics remains under-explored. We present CricBench, a benchmark suite evaluating the intrinsic SQL generation abilities of LLMs on cricket data across four formats: Test, ODI, T20I, and IPL. We curate a Gold-Standard dataset of 2,654 evaluation instances across four languages (English, Hindi, Punjabi, and Telugu). We evaluate seven models, GPT-5 Mini, Claude Sonnet 4, DeepSeek R1 and V3, Qwen 235B, Llama 3.1, and Gemma 2, using schema-only prompting. No single model dominates across all formats: GPT-5 Mini leads on Test cricket (12.4% DMA), Qwen 235B leads on IPL (28.7%) and T20I (17.5%), and all models score 0% on hard ODI queries. All models show a stark disconnect between syntactic validity (>98% execution accuracy) and semantic correctness (<29% DMA), with a domain gap of 37-55 percentage points versus BIRD. To our knowledge, CricBench is the first Text-to-SQL benchmark for cricket analytics.