UbuntuGuard: A Culturally-Grounded Policy Benchmark for Equitable AI Safety in African Languages

TL;DR

UbuntuGuard introduces a culturally grounded safety benchmark for African languages, addressing the limitations of Western-centric models and emphasizing the importance of local norms and policies for equitable AI safety.

Contribution

It presents the first policy-based safety benchmark for African languages, derived from expert-crafted queries, to evaluate and improve culturally aligned guardian models.

Findings

Existing benchmarks overestimate multilingual safety in African languages.

Cross-lingual transfer offers partial coverage but is insufficient.

Dynamic models better leverage safety policies but still struggle with local contexts.

Abstract

Current guardian models are predominantly Western-centric and optimized for high-resource languages, leaving low-resource African languages vulnerable to evolving harms, cross-lingual failures, and cultural misalignment. Moreover, most guardian models rely on rigid, predefined safety categories that fail to generalize across diverse linguistic and sociocultural contexts. Achieving robust safety requires flexible, runtime-enforceable policies and benchmarks that reflect local norms, harm scenarios, and cultural expectations. We introduce UbuntuGuard, the first policy-based safety benchmark for African languages built from adversarial queries authored by 155 domain experts across sensitive fields, including healthcare. From these expert-crafted queries, we derive context-specific safety policies and reference responses that capture culturally grounded risk signals, enabling policy-aligned evaluation of guardian models. We evaluate 15 models, comprising seven general-purpose LLMs and eight guardian models across three distinct variants: static, dynamic, and multilingual. Our findings reveal that existing English-centric benchmarks overestimate real-world multilingual safety, cross-lingual transfer provides partial but insufficient coverage, and dynamic models, while better equipped to leverage policies at inference time, still struggle to fully localize African-language contexts. These findings highlight the urgent need for multilingual, culturally grounded safety benchmarks to enable the development of reliable and equitable guardian models for low-resource languages.