IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text
Rajveer Singh Pall

TL;DR
IndiaFinBench is the first publicly available benchmark for evaluating large language models on Indian financial regulatory texts, covering interpretation, reasoning, and contradiction detection tasks.
Contribution
It introduces a novel dataset with expert annotations from Indian financial regulators, filling a gap in non-Western financial NLP benchmarks.
Findings
Models outperform non-specialist humans on the benchmark.
Numerical reasoning tasks show the largest performance gap among models.
Three performance tiers are statistically distinguished through bootstrap testing.
Abstract
We introduce IndiaFinBench, to our knowledge the first publicly available evaluation benchmark for assessing large language model (LLM) performance on Indian financial regulatory text. Existing financial NLP benchmarks draw exclusively from Western financial corpora (SEC filings, US earnings reports, English-language financial news), leaving a significant gap in coverage of non-Western regulatory frameworks. IndiaFinBench addresses this gap with 406 expert-annotated question-answer pairs drawn from 192 documents sourced from the Securities and Exchange Board of India (SEBI) and the Reserve Bank of India (RBI), spanning four task types: regulatory interpretation (174 items), numerical reasoning (92 items), contradiction detection (62 items), and temporal reasoning (78 items). Annotation quality is validated through a model-based secondary pass (kappa=0.918 on contradiction detection;…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
