SalamahBench: Toward Standardized Safety Evaluation for Arabic Language Models

TL;DR

This paper introduces SalamaBench, a comprehensive safety evaluation benchmark for Arabic Language Models, addressing the lack of Arabic-specific safety assessments and revealing significant safety vulnerabilities across models.

Contribution

It presents SalamaBench, a novel, category-aware safety benchmark for Arabic LMs, and evaluates multiple models, highlighting safety gaps and the importance of specialized safeguards.

Findings

Fanar 2 has the lowest attack success rates but uneven safety across domains.

Jais 2 shows higher vulnerability and weaker safety alignment.

Native ALMs underperform compared to safeguard models as safety judges.

Abstract

Safety alignment in Language Models (LMs) is fundamental for trustworthy AI. However, while different stakeholders are trying to leverage Arabic Language Models (ALMs), systematic safety evaluation of ALMs remains largely underexplored, limiting their mainstream uptake. Existing safety benchmarks and safeguard models are predominantly English-centric, limiting their applicability to Arabic Natural Language Processing (NLP) systems and obscuring fine-grained, category-level safety vulnerabilities. This paper introduces SalamaBench, a unified benchmark for evaluating the safety of ALMs, comprising $8,170$ prompts across $12$ different categories aligned with the MLCommons Safety Hazard Taxonomy. Constructed by harmonizing heterogeneous datasets through a rigorous pipeline involving AI filtering and multi-stage human verification, SalamaBench enables standardized, category-aware safety evaluation. Using this benchmark, we evaluate five state-of-the-art ALMs, including Fanar 1 and 2, ALLaM 2, Falcon H1R, and Jais 2, under multiple safeguard configurations, including individual guard models, majority-vote aggregation, and validation against human-annotated gold labels. Our results reveal substantial variation in safety alignment: while Fanar 2 achieves the lowest aggregate attack success rates, its robustness is uneven across specific harm domains. In contrast, Jais 2 consistently exhibits elevated vulnerability, indicating weaker intrinsic safety alignment. We further demonstrate that native ALMs perform substantially worse than dedicated safeguard models when acting as safety judges. Overall, our findings highlight the necessity of category-aware evaluation and specialized safeguard mechanisms for robust harm mitigation in ALMs.