SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning

TL;DR

SAHM is the first Arabic financial benchmark with seven tasks, designed to evaluate and improve Arabic financial reasoning and compliance in NLP models, supporting trustworthy financial AI tools.

Contribution

Introduces Sahm, a comprehensive Arabic financial NLP benchmark with 14,380 instances across seven tasks, filling a critical gap in Arabic financial AI research.

Findings

Arabic fluency does not guarantee financial reasoning ability in models.

Models perform well on recognition but poorly on generation tasks.

Event-cause reasoning shows the largest performance gap.

Abstract

English financial NLP has advanced rapidly through benchmarks targeting earnings analysis, market sentiment, tabular reasoning, and financial question answering, yet Arabic financial NLP remains virtually nonexistent, despite 422 million speakers, $4.9 trillion in Gulf sovereign wealth, and a $4-5 trillion Islamic finance industry requiring specialized Shari'ah compliance over instruments like sukuk, murabaha, and takaful. We introduce Sahm, the first Arabic financial benchmark spanning seven tasks: AAOIFI standards QA, fatwa-based QA/MCQ, accounting and business exams, financial sentiment analysis, extractive summarization, and event-cause reasoning, comprising 14,380 expert-verified instances from authentic regulatory, juristic, and corporate sources. Evaluating 20 LLMs, we find Arabic fluency does not imply financial reasoning: models achieving 91% on recognition tasks drop sharply on generation, and event-cause reasoning exposes the widest performance gap (1.89-9.84/10). We release the benchmark and dataset to support trustworthy Arabic financial assistants.