Efficient Elicitation of Collective Disagreements

TL;DR

This paper introduces a stratified framework and the plurality matrix to efficiently measure and analyze disagreements among voters, highlighting the limitations of pairwise comparisons and proposing protocols for better preference elicitation.

Contribution

It proposes a new plurality matrix concept and a stratified framework to identify minimal preference information needed for disagreement measures, advancing beyond traditional pairwise comparisons.

Findings

Many disagreement measures are at level 3, requiring more than pairwise comparisons.

Going beyond level 3 provides better insights into voter disagreements.

Protocols are designed to balance participant number and cognitive load.

Abstract

We analyze the structure of the disagreement among a population of voters over a set of alternatives. Surveys typically ask either for pairwise comparisons, simple and intuitive for participants, or full rankings over alternatives, eliciting the entire voters' preferences. Building on the observation that pairwise comparisons cannot distinguish structural disagreement from noise, we propose a stratified framework to identify the minimal aggregated preference information needed to compute a number of disagreement measures from the literature. Specifically, we introduce the plurality matrix, a generalization of pairwise comparisons that records, for every subset $S$ of alternatives, the probability that each $a \in S$ ranks first in $S$. We define the level of a disagreement measure as the smallest subset size needed to express it, showing that many existing notions, including rank-variance and divisiveness, sit at level $3$, proving that pairwise comparisons are not enough. In addition, we demonstrate the interest of going beyond level $3$ both theoretically and experimentally. To make these results actionable, we design two elicitation protocols to estimate the plurality matrix, exploring the trade-off between the number of required participants and the cognitive load requested to each of them.