Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset

TL;DR

This paper introduces the Community Alignment dataset, a large multilingual preference dataset, and demonstrates the importance of diverse sampling methods to better align large language models with varied human preferences across cultures.

Contribution

It presents a novel negatively-correlated sampling technique for preference data collection and releases the largest multilingual preference dataset to date.

Findings

Humans show more preference variation than LLM responses.

Existing datasets are insufficient for capturing global preference diversity.

Negatively-correlated sampling improves preference learning.

Abstract

How can large language models (LLMs) serve users with varying preferences that may conflict across cultural, political, or other dimensions? To advance this challenge, this paper establishes four key results. First, we demonstrate, through a large-scale multilingual human study with representative samples from five countries (N=15,000), that humans exhibit substantially more variation in preferences than the responses of 21 state-of-the-art LLMs. Second, we show that existing methods for preference dataset collection are insufficient for learning the diversity of human preferences even along two of the most salient dimensions of variability in global values, due to the underlying homogeneity of candidate responses. Third, we argue that this motivates the need for negatively-correlated sampling when generating candidate sets, and we show that simple prompt-based techniques for doing so greatly enhance the performance of alignment methods in learning heterogeneous preferences. Fourth, based on this novel candidate sampling approach, we collect and open-source Community Alignment} the largest and most representative multilingual and multi-turn preference dataset to date, featuring 233,319 comparisons from annotators spanning five countries. Overall, we hope that the Community Alignment dataset will be a valuable resource for improving the effectiveness of LLMs for a diverse global population.