CultureGuard: Towards Culturally-Aware Dataset and Guard Model for Multilingual Safety Applications

TL;DR

CultureGuard introduces a pipeline for creating culturally aligned safety datasets in multiple languages, enabling the training of multilingual safety guard models that outperform existing benchmarks and generalize well across languages.

Contribution

We propose a novel four-stage pipeline for generating high-quality, culturally aligned safety datasets across multiple languages, facilitating multilingual safety guard model development.

Findings

The dataset contains 386,661 samples across 9 languages.

The trained model achieves state-of-the-art safety benchmark performance.

Multilingual fine-tuning improves cross-lingual transfer and zero-shot generalization.

Abstract

The increasing use of Large Language Models (LLMs) in agentic applications highlights the need for robust safety guard models. While content safety in English is well-studied, non-English languages lack similar advancements due to the high cost of collecting culturally aligned labeled datasets. We present CultureGuard, a novel solution for curating culturally aligned, high-quality safety datasets across multiple languages. Our approach introduces a four-stage synthetic data generation and filtering pipeline: cultural data segregation, cultural data adaptation, machine translation, and quality filtering. This pipeline enables the conversion and expansion of the Nemotron-Content-Safety-Dataset-V2 English safety dataset into eight distinct languages: Arabic, German, Spanish, French, Hindi, Japanese, Thai, and Chinese. The resulting dataset, Nemotron-Safety-Guard-Dataset-v3, comprises 386,661 samples in 9 languages and facilitates the training of Llama-3.1-Nemotron-Safety-Guard-8B-v3 via LoRA-based fine-tuning. The final model achieves state-of-the-art performance on several multilingual content safety benchmarks. Furthermore, we show our moderately multilingual fine-tuning enables robust cross-lingual transfer and strong zero-shot generalization to unseen languages. We also benchmark the latest open LLMs on multilingual safety and observe that these LLMs are more prone to give unsafe responses when prompted in non-English languages. This work advances multilingual LLM safety by enabling the development of culturally aware safety guard models.