Question the Questions: Auditing Representation in Online Deliberative Processes

TL;DR

This paper introduces an auditing framework for measuring question representativeness in online deliberative processes, providing algorithms and applying them to real-world data, including LLM-generated questions.

Contribution

It presents the first algorithms for auditing justified representation in general utility settings and demonstrates their application to historical and LLM-generated questions.

Findings

Algorithms efficiently audit question representativeness with $O(mn\,\log n)$ runtime.

LLMs show promise but have limitations in supporting deliberative processes.

Auditing methods can improve question selection for better representation.

Abstract

A central feature of many deliberative processes, such as citizens' assemblies and deliberative polls, is the opportunity for participants to engage directly with experts. While participants are typically invited to propose questions for expert panels, only a limited number can be selected due to time constraints. This raises the challenge of how to choose a small set of questions that best represent the interests of all participants. We introduce an auditing framework for measuring the level of representation provided by a slate of questions, based on the social choice concept known as justified representation (JR). We present the first algorithms for auditing JR in the general utility setting, with our most efficient algorithm achieving a runtime of $O(mn\log n)$, where $n$ is the number of participants and $m$ is the number of proposed questions. We apply our auditing methods to historical deliberations, comparing the representativeness of (a) the actual questions posed to the expert panel (chosen by a moderator), (b) participants' questions chosen via integer linear programming, (c) summary questions generated by large language models (LLMs). Our results highlight both the promise and current limitations of LLMs in supporting deliberative processes. By integrating our methods into an online deliberation platform that has been used for over hundreds of deliberations across more than 50 countries, we make it easy for practitioners to audit and improve representation in future deliberations.