UnWEIRDing LLM Entity Recommendations

TL;DR

This paper investigates cultural biases in Large Language Model (LLM) entity recommendations using the WEIRD framework, demonstrating that prompt-based strategies can mitigate biases but with inconsistent effectiveness across models and entity types.

Contribution

It introduces a systematic evaluation of cultural biases in LLM recommendations for real-world entities and explores prompt-based mitigation strategies.

Findings

Prompt strategies reduce biases but inconsistently across models.

Bias varies significantly across different entity types.

Some models exhibit more cultural bias than others.

Abstract

Large Language Models have been widely been adopted by users for writing tasks such as sentence completions. While this can improve writing efficiency, prior research shows that LLM-generated suggestions may exhibit cultural biases which may be difficult for users to detect, especially in educational contexts for non-native English speakers. While such prior work has studied the biases in LLM moral value alignment, we aim to investigate cultural biases in LLM recommendations for real-world entities. To do so, we use the WEIRD (Western, Educated, Industrialized, Rich and Democratic) framework to evaluate recommendations by various LLMs across a dataset of fine-grained entities, and apply pluralistic prompt-based strategies to mitigate these biases. Our results indicate that while such prompting strategies do reduce such biases, this reduction is not consistent across different models, and recommendations for some types of entities are more biased than others.