DeliberationBench: A Normative Benchmark for the Influence of Large Language Models on Users' Views

TL;DR

This paper introduces DeliberationBench, a benchmark for evaluating the normative influence of large language models on users' beliefs through deliberative opinion polling, demonstrating that LLMs can have broadly beneficial effects on opinion shifts.

Contribution

It proposes a normative benchmark for assessing LLM influence on beliefs, integrating deliberative polling to distinguish beneficial from harmful influence.

Findings

LLMs significantly influence user opinions in deliberative settings.

Influence correlates positively with net opinion shifts after deliberation.

Framework can monitor LLM influence to ensure democratic legitimacy.

Abstract

As large language models (LLMs) become pervasive as assistants and thought partners, it is important to characterize their persuasive influence on users' beliefs. However, a central challenge is to distinguish "beneficial" from "harmful" forms of influence, in a manner that is normatively defensible and legitimate. We propose DeliberationBench, a benchmark for assessing LLM influence that takes the process of deliberative opinion polling as its standard. We demonstrate our approach in a preregistered randomized experiment in which 4,088 U.S. participants discussed 65 policy proposals with six frontier LLMs. Using opinion change data from four prior Deliberative Polls conducted by the Deliberative Democracy Lab, we find evidence that the tested LLMs' influence is substantial in magnitude and positively associated with the net opinion shifts following deliberation, suggesting that these models exert broadly epistemically desirable effects. We further explore differential influence between topic areas, demographic subgroups, and models. Our framework can function as an evaluation and monitoring tool, helping to ensure that the influence of LLMs remains consistent with democratically legitimate standards, and preserves users' autonomy in forming their views.