CAGE: A Framework for Culturally Adaptive Red-Teaming Benchmark Generation

TL;DR

CAGE is a framework that systematically adapts red-teaming prompts to different cultural contexts, improving the detection of socio-technical vulnerabilities in language models beyond simple translation methods.

Contribution

It introduces the Semantic Mold approach for disentangling adversarial structure from cultural content, enabling realistic, localized threat modeling in safety benchmarks.

Findings

KoRSET outperforms translation baselines in vulnerability detection.

CAGE provides scalable, culturally-aware safety benchmarks.

The dataset and evaluation tools are publicly available.

Abstract

Existing red-teaming benchmarks, when adapted to new languages via direct translation, fail to capture socio-technical vulnerabilities rooted in local culture and law, creating a critical blind spot in LLM safety evaluation. To address this gap, we introduce CAGE (Culturally Adaptive Generation), a framework that systematically adapts the adversarial intent of proven red-teaming prompts to new cultural contexts. At the core of CAGE is the Semantic Mold, a novel approach that disentangles a prompt's adversarial structure from its cultural content. This approach enables the modeling of realistic, localized threats rather than testing for simple jailbreaks. As a representative example, we demonstrate our framework by creating KoRSET, a Korean benchmark, which proves more effective at revealing vulnerabilities than direct translation baselines. CAGE offers a scalable solution for developing meaningful, context-aware safety benchmarks across diverse cultures. Our dataset and evaluation rubrics are publicly available at https://github.com/selectstar-ai/CAGE-paper. (WARNING: This paper contains model outputs that can be offensive in nature.)