LSR: Linguistic Safety Robustness Benchmark for Low-Resource West African Languages

TL;DR

This paper introduces LSR, a benchmark for measuring how well large language models maintain safety refusal behaviors across West African languages, revealing significant degradation compared to English.

Contribution

It presents the first systematic cross-lingual safety benchmark for West African languages and introduces the Refusal Centroid Drift metric to quantify refusal degradation.

Findings

English refusal rates are around 90% in models.

Refusal rates drop to 35-55% in West African languages.

Igala shows the most severe refusal degradation (RCD = 0.55).

Abstract

Safety alignment in large language models relies predominantly on English-language training data. When harmful intent is expressed in low-resource languages, refusal mechanisms that hold in English frequently fail to activate. We introduce LSR (Linguistic Safety Robustness), the first systematic benchmark for measuring cross-lingual refusal degradation in West African languages: Yoruba, Hausa, Igbo, and Igala. LSR uses a dual-probe evaluation protocol - submitting matched English and target-language probes to the same model - and introduces Refusal Centroid Drift (RCD), a metric that quantifies how much of a model's English refusal behavior is lost when harmful intent is encoded in a target language. We evaluate Gemini 2.5 Flash across 14 culturally grounded attack probes in four harm categories. English refusal rates hold at approximately 90 percent. Across West African languages, refusal rates fall to 35-55 percent, with Igala showing the most severe degradation (RCD = 0.55). LSR is implemented in the Inspect AI evaluation framework and is available as a PR-ready contribution to the UK AISI's inspect_evals repository. A live reference implementation and the benchmark dataset are publicly available.