CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models

TL;DR

This paper presents CIVICS, a multilingual dataset of value-sensitive prompts designed to evaluate how large language models respond to culturally and socially sensitive topics across different languages and contexts.

Contribution

The paper introduces a novel, handcrafted multilingual dataset for assessing LLMs' responses to culturally-informed, value-laden prompts, enabling analysis of social and cultural variability in model behavior.

Findings

LLMs show variability in responses across languages and topics.

Refusal rates differ by language and translation.

Certain topics elicit more pronounced differences in responses.

Abstract

This paper introduces the "CIVICS: Culturally-Informed & Values-Inclusive Corpus for Societal impacts" dataset, designed to evaluate the social and cultural variation of Large Language Models (LLMs) across multiple languages and value-sensitive topics. We create a hand-crafted, multilingual dataset of value-laden prompts which address specific socially sensitive topics, including LGBTQI rights, social welfare, immigration, disability rights, and surrogacy. CIVICS is designed to generate responses showing LLMs' encoded and implicit values. Through our dynamic annotation processes, tailored prompt design, and experiments, we investigate how open-weight LLMs respond to value-sensitive issues, exploring their behavior across diverse linguistic and cultural contexts. Using two experimental set-ups based on log-probabilities and long-form responses, we show social and cultural variability across different LLMs. Specifically, experiments involving long-form responses demonstrate that refusals are triggered disparately across models, but consistently and more frequently in English or translated statements. Moreover, specific topics and sources lead to more pronounced differences across model answers, particularly on immigration, LGBTQI rights, and social welfare. As shown by our experiments, the CIVICS dataset aims to serve as a tool for future research, promoting reproducibility and transparency across broader linguistic settings, and furthering the development of AI technologies that respect and reflect global cultural diversities and value pluralism. The CIVICS dataset and tools will be made available upon publication under open licenses; an anonymized version is currently available at https://huggingface.co/CIVICS-dataset.