Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Prompting

TL;DR

This study evaluates whether socio-demographic prompts genuinely influence large language models' responses or if observed effects are due to random variations, revealing limitations in current cultural bias probing methods.

Contribution

It systematically tests multiple LLMs with culturally conditioned prompts, highlighting the potential placebo effect and questioning the robustness of such bias elicitation techniques.

Findings

Most models show response variation with prompts, except GPT-4.

Cultural prompting effects may be due to placebo effects rather than true bias.

Calls for improved experimental design to distinguish genuine bias from random response changes.

Abstract

Socio-demographic prompting is a commonly employed approach to study cultural biases in LLMs as well as for aligning models to certain cultures. In this paper, we systematically probe four LLMs (Llama 3, Mistral v0.2, GPT-3.5 Turbo and GPT-4) with prompts that are conditioned on culturally sensitive and non-sensitive cues, on datasets that are supposed to be culturally sensitive (EtiCor and CALI) or neutral (MMLU and ETHICS). We observe that all models except GPT-4 show significant variations in their responses on both kinds of datasets for both kinds of prompts, casting doubt on the robustness of the culturally-conditioned prompting as a method for eliciting cultural bias in models or as an alignment strategy. The work also calls rethinking the control experiment design to tease apart the cultural conditioning of responses from "placebo effect", i.e., random perturbations of model responses due to arbitrary tokens in the prompt.