LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models

TL;DR

LinguaSafe introduces a large, diverse multilingual safety benchmark with 45,000 entries across 12 languages, enabling more comprehensive safety evaluations of large language models in various linguistic and cultural contexts.

Contribution

This work provides the first extensive multilingual safety benchmark with detailed evaluation metrics, addressing the lack of diverse safety assessments for under-represented languages.

Findings

Safety and helpfulness vary across languages and domains.

Multilingual safety assessments reveal significant differences even among resource-similar languages.

The benchmark facilitates more balanced safety alignment in LLMs.

Abstract

The widespread adoption and increasing prominence of large language models (LLMs) in global technologies necessitate a rigorous focus on ensuring their safety across a diverse range of linguistic and cultural contexts. The lack of a comprehensive evaluation and diverse data in existing multilingual safety evaluations for LLMs limits their effectiveness, hindering the development of robust multilingual safety alignment. To address this critical gap, we introduce LinguaSafe, a comprehensive multilingual safety benchmark crafted with meticulous attention to linguistic authenticity. The LinguaSafe dataset comprises 45k entries in 12 languages, ranging from Hungarian to Malay. Curated using a combination of translated, transcreated, and natively-sourced data, our dataset addresses the critical need for multilingual safety evaluations of LLMs, filling the void in the safety evaluation of LLMs across diverse under-represented languages from Hungarian to Malay. LinguaSafe presents a multidimensional and fine-grained evaluation framework, with direct and indirect safety assessments, including further evaluations for oversensitivity. The results of safety and helpfulness evaluations vary significantly across different domains and different languages, even in languages with similar resource levels. Our benchmark provides a comprehensive suite of metrics for in-depth safety evaluation, underscoring the critical importance of thoroughly assessing multilingual safety in LLMs to achieve more balanced safety alignment. Our dataset and code are released to the public to facilitate further research in the field of multilingual LLM safety.