Evidence-based Distributional Alignment for Large Language Models

TL;DR

Evi-DA is a novel method that enhances large language models' ability to predict accurate, culturally-aware answer distributions by leveraging survey data and reinforcement learning, outperforming existing approaches especially under domain shifts.

Contribution

The paper introduces Evi-DA, a new evidence-based alignment technique that improves distributional predictions of LLMs using survey data and reinforcement learning, addressing instability and bias issues.

Findings

Reduces Jensen-Shannon divergence by up to 44%

Improves robustness under domain and cultural shifts

Enhances distribution fidelity across benchmarks

Abstract

Distributional alignment enables large language models (LLMs) to predict how a target population distributes its responses across answer options, rather than collapsing disagreement into a single consensus answer. However, existing LLM-based distribution prediction is often unstable and degrades under cultural and domain shift. Token score-based estimates can change with minor option wording or formatting, response sampling-based estimates are expensive and sensitive to prompts and decoding settings, and directly generated distributions are frequently miscalibrated. We propose Evi-DA, an evidence-based alignment technique that improves the fidelity and robustness of LLM-based distribution estimation under domain and cultural shift. Given a target country and a multiple-choice question, Evi-DA retrieves related World Values Survey items and their answer distributions, predicts a coarse Welzel value signature for each option, and infers the country-conditioned answer distribution in a structured format. We train the LLMs using a two-stage pipeline, where reinforcement learning optimizes survey-derived rewards that encourage accurate intermediate value predictions, faithful final distributions, well-formed structured outputs, and reduced cultural bias. Across in-domain and out-of-domain benchmarks and multiple open-source backbones, Evi-DA reduces Jensen-Shannon divergence between predicted and gold distributions relative to strong baselines, with average relative improvements of up to 44%.