Do Large Language Models Reflect Demographic Pluralism in Safety?

TL;DR

This paper introduces Demo-SafetyBench, a new benchmark for evaluating large language models' safety across diverse demographic perspectives, highlighting the importance of pluralism in safety assessments.

Contribution

It models demographic diversity directly at the prompt level and evaluates LLM safety sensitivity across communities, addressing limitations of prior demographically narrow datasets.

Findings

High reliability (ICC = 0.87) in safety evaluation across demographics

Low demographic sensitivity (DS = 0.12) achieved with balanced thresholds

Scalable safety evaluation method incorporating demographic pluralism

Abstract

Large Language Model (LLM) safety is inherently pluralistic, reflecting variations in moral norms, cultural expectations, and demographic contexts. Yet, existing alignment datasets such as ANTHROPIC-HH and DICES rely on demographically narrow annotator pools, overlooking variation in safety perception across communities. Demo-SafetyBench addresses this gap by modeling demographic pluralism directly at the prompt level, decoupling value framing from responses. In Stage I, prompts from DICES are reclassified into 14 safety domains (adapted from BEAVERTAILS) using Mistral 7B-Instruct-v0.3, retaining demographic metadata and expanding low-resource domains via Llama-3.1-8B-Instruct with SimHash-based deduplication, yielding 43,050 samples. In Stage II, pluralistic sensitivity is evaluated using LLMs-as-Raters-Gemma-7B, GPT-4o, and LLaMA-2-7B-under zero-shot inference. Balanced thresholds (delta = 0.5, tau = 10) achieve high reliability (ICC = 0.87) and low demographic sensitivity (DS = 0.12), confirming that pluralistic safety evaluation can be both scalable and demographically robust.