Who is in Your Top Three? Optimizing Learning in Elections with Many Candidates

TL;DR

This paper develops a theoretical framework for optimizing voting mechanisms in elections with many candidates, focusing on how to choose the number of top candidates to consider and the aggregation rule to maximize learning efficiency.

Contribution

It introduces a method to select the optimal K-Approval or K-partial ranking mechanism based on voter distribution, improving election outcome accuracy and speed.

Findings

Optimal mechanisms significantly improve winner identification probability.

Historical elections often set K too low, reducing accuracy.

Data-driven approach enhances election outcome certainty from 80% to 99.9%.

Abstract

Elections and opinion polls often have many candidates, with the aim to either rank the candidates or identify a small set of winners according to voters' preferences. In practice, voters do not provide a full ranking; instead, each voter provides their favorite K candidates, potentially in ranked order. The election organizer must choose K and an aggregation rule. We provide a theoretical framework to make these choices. Each K-Approval or K-partial ranking mechanism (with a corresponding positional scoring rule) induces a learning rate for the speed at which the election correctly recovers the asymptotic outcome. Given the voter choice distribution, the election planner can thus identify the rate optimal mechanism. Earlier work in this area provides coarse order-of-magnitude guaranties which are not sufficient to make such choices. Our framework further resolves questions of when randomizing between multiple mechanisms may improve learning, for arbitrary voter noise models. Finally, we use data from 5 large participatory budgeting elections that we organized across several US cities, along with other ranking data, to demonstrate the utility of our methods. In particular, we find that historically such elections have set K too low and that picking the right mechanism can be the difference between identifying the ultimate winner with only a 80% probability or a 99.9% probability after 400 voters.

Tables

Table 1. Table 1: List of election data that we use in Section  6 . From PrefLib, we use all elections where full rankings are available and there are at least 5 candidates and 700 voters. Throughout, we ignore voters who did not submit full rankings (especially with high K 𝐾 K -Ranking requested, this might only be a fraction of the total number of actual votes). Additionally, for the PB elections, we limit the data to those who submitted votes online rather than through paper ballots.   Sources for the PrefLib datasets are: Mattei and Walsh ( 2013 ); O’Neill ( 2013 ); Regenwetter et al . ( 2007 , 2008 ); Popov, Popova, and Regenwetter ( 2014 ) .

Name	Candidates	Votes with complete rankings	K-Ranking available
Participatory Budgeting
Boston, 2016	8	4173	4
Durham Ward 1, 2019	21	1637	10
Durham Ward 2, 2019	10	329	10
Durham Ward 3, 2019	12	694	10
Rochester, 2019	22	649	5
PrefLib:
Irish01	12	4259	12
Irish02	9	4810	9
Irish03	14	3166	14
ElectorialReformSociety77	12	1312	12
ElectorialReformSociety13	5	1809	5
Sushi10	10	5000	10
Glasgow05	10	718	10
Glasgow17	9	962	9
Glasgow10	9	818	9
Glasgow18	9	767	9
Glasgow20	9	726	9
Glasgow14	8	1071	8
Glasgow12	8	1040	8
Burlington01	6	2603	6
Burlington02	6	2853	6
APA03	5	11539	5
APA01	5	10978	5
APA11	5	10791	5
APA05	5	10655	5
APA02	5	10623	5
APA04	5	10519	5
APA09	5	10211	5
APA06	5	10177	5
APA07	5	9747	5
APA12	5	9091	5
APA08	5	8532	5
APA10	5	8467	5
Aspen02	5	1183	5

Table 2. Table 2 : OLS Regression on the best K to use in K-Approval, by the number of candidates and desired winners. Standard errors are cluster standard errors, where each cluster is an election in our dataset.

Dep. Variable:	Best Mechanism	R-squared:	0.273
Model:	OLS	Adj. R-squared:	0.264
Method:	Least Squares	F-statistic:	42.68
Date:	Wed, 12 Jun 2019	Prob (F-statistic):	2.72e-11
Time:	16:17:32	Log-Likelihood:	-531.30
No. Observations:	241	AIC:	1071.
Df Residuals:	237	BIC:	1085.
Df Model:	3

Table 3. Table 3 : Elections and goals where randomizing between two K 𝐾 K -Approval mechanisms produces leads to faster learning than using either of the mechanisms separately. For several of these cases, randomization also beats the Approval rate optimal mechanism.

Election	Number Winners	Mechanism 1	Mechanism 2	Beats Approval rate optimal
Durham Ward 1, 2019	2	3	4	True
Durham Ward 1, 2019	13	8	9	True
Durham Ward 1, 2019	17	6	7	True
Irish03	1	1	3	False
Irish03	10	8	9	False
Irish03	10	8	10	False
Irish01	5	3	4	True
Irish01	5	3	5	False
Irish01	5	3	11	False
Irish01	7	1	5	False
Glasgow05	2	4	5	True
Glasgow05	2	5	6	True
Glasgow10	2	3	4	False
Glasgow10	2	4	6	False
Glasgow10	2	5	6	True
APA08	2	2	3	True

	Voter 1	Voter 2	Voter 3	Voter 4	Voter 5
Rank 1	A	A	D	D	B
Rank 2	B	B	C	C	C
Rank 3	C	C	B	B	D
Rank 4	D	D	A	A	A

Table 5. Table 4 : Where the constants such that 0 < ϵ < a < T 2 2 < T 2 < T 1 < T 1 + 2 ​ T 2 + a = 1 0 italic-ϵ 𝑎 subscript 𝑇 2 2 subscript 𝑇 2 subscript 𝑇 1 subscript 𝑇 1 2 subscript 𝑇 2 𝑎 1 0<\epsilon<a<\frac{T_{2}}{2}<T_{2}<T_{1}<T_{1}+2T_{2}+a=1 , i.e., the table describes a valid probability distribution.Row 10 is as follows: The first K 𝐾 K candidates (the asymptotic winners) occupy the first K 𝐾 K spots, in an order drawn uniformly at random. Similarly, The bottom M − K 𝑀 𝐾 M-K candidates occupy the bottom M − K 𝑀 𝐾 M-K spots, in an order drawn uniformly at random. This randomization ensures asymptotically design invariance.

Positions of $h,i,j$

Contributes to? (Y = Yes)

Row

$P{r}_F(\cdot )$

$\sigma \left(h\right)$

$\sigma \left(i\right)$

$\sigma \left(j\right)$

$t_{hi}^h\left(K\right)$

$t_{hi}^i\left(K\right)$

$t_{hi}^h\left(L\right)$

$t_{hi}^i\left(L\right)$

$t_{hj}^h\left(K\right)$

$t_{hj}^j\left(K\right)$

$t_{hj}^h\left(L\right)$

$t_{hj}^j\left(L\right)$

1

$a$

$K$

$L+1$

$L$

Y

Y

Y

2

$a$

$K-1$

$K$

$L$

Y

3

$T_1-a-ϵ$

$K$

$L+1$

$L+2$

Y

Y

Y

Y

4

$T_2-2a$

$L+1$

$L+2$

$K$

Y

Y

5

$T_2-2a$

$L+1$

$K$

$L+2$

Y

Y

6

$a$

$L+1$

$K$

$L$

Y

Y

Y

7

$a$

$L$

$K-1$

$K$

Y

Y

8

$a$

$L$

$L+1$

$L+2$

Y

Y

9

$a$

$L+1$

$L+2$

L

Y

10

$ϵ$

See caption

Y

Y

Y

Y

11

$0$

Otherwise

Total value:

$T_1$

$T_2$

$T_1+a$

$T_2-a$

$T_1+a$

$T_2-a$

$T_1$

$T_2$

Candidate		$K-1$	$K$	$L=K+1$	$K+2$	$K+3$	$M>k>K+3$
$w\in \left\{1\dots K-2\right\}$	$>0$	$>1-ϵ$	$1$	$1$	$1$	$1$	$1$
$K-1$	$>0$	$1-2a$	$1-2a$	$1-2a$	$1$	$1$	$1$
$K$	$>0$	$>a$	$T_1+a$	$T_1+3a$	$1$	$1$	$1$
$L=K+1$	$0$	$a$	$T_2$
$K+2$	$0$	$0$	$T_2-a$
$q\in \left\{K+3\dots M\right\}$	$0$	$0$	$\frac{2a}{Q}$	$\frac{T_1+2T_2-3a-ϵ}{Q}$

Equations

$$F_{\text{Mallows}}(\sigma )\propto {\varphi }^{d(\sigma ,{\sigma }^{\ast })}$$

$$\underset{N\to \infty }{\lim }{O}^N(M,F,\beta ,G)=O^{\ast },\text{ with probability }1$$

$$r=-\underset{N\to \infty }{\lim }\frac{1}{N}\log A_N$$

$$r_{ij}(\beta )=-\underset{z\in \mathbb{R}}{\inf }\log {\mathbb{E}}_F\left[\exp \left(z\left(\beta ({\sigma }_v(i))-\beta ({\sigma }_v(j))\right)\right)\right]$$

$$\text{Pr}({\sigma }^N(i)>{\sigma }^N(j))\le \exp (-r_{ij}(\beta )N)$$

$$r_{ij}(K)$$

$$r(\beta )=\underset{i\in C_s^{\ast },j\in C_t^{\ast },s<t}{\min }{r}_{ij}(\beta )$$

$$Q^N\le M^2\exp (-rN)$$

$$r(B,D)>\underset{K}{\max }r(K)$$

$$\forall i\in C:\mathbb{E}[s_{iv}]$$

$$=\sum _{m=1}^M\beta (m){\text{Pr}}_F({\sigma }_v(i)\le m)-\sum _{m=2}^M\beta (m){\text{Pr}}_F({\sigma }_v(i)<m)$$

$$=\sum _{m=1}^M\beta (m){\text{Pr}}_F({\sigma }_v(i)\le m)-\sum _{m=1}^{M-1}\beta (m+1){\text{Pr}}_F({\sigma }_v(i)\le m)$$

$$=\beta (M)+\sum _{m=1}^{M-1}\left[\beta (m)-\beta (m+1)\right]{\text{Pr}}_F({\sigma }_v(i)\le m)$$

$$\exists O^{\ast }:\forall \beta \in \mathcal{B},\underset{N\to \infty }{\lim }\mathcal{O}(M,N,F,\beta ,G)=O^{\ast },\text{ with probability }1$$

$$\beta (m)=\{\begin{array}{ll}1& m\le k\\ 0& m>k\end{array}$$

$$\underset{N\to \infty }{\lim }\mathcal{O}(M,N,F,\beta ,G)\ne O^{\ast }$$

$$\mathbb{E}[s_{iv}]$$

$$>\mathbb{E}[s_{jv}]$$

$$A_v=\beta ({\sigma }_v(i))-\beta ({\sigma }_v(j))$$

$$r_{ij}(\beta )$$

$$\Lambda (z)$$

$$-\underset{N\to \infty }{\lim }\frac{1}{N}\log \text{Pr}(A^N\le 0)$$

$$\text{Pr}({\sigma }^N(i)>{\sigma }^N(j))$$

$$<{\left[\underset{z>0}{\inf }\mathbb{E}[\exp [-z{A}_v]]\right]}^N$$

$$={\left[\underset{z<0}{\inf }\left[\sum _{m=1}^M\sum _{\ell \ne m}\text{Pr}({\sigma }_v(i)=m,{\sigma }_v(j)=\ell )\exp [z(\beta ({\sigma }_v(i))-\beta ({\sigma }_v(j)))]\right]\right]}^N$$

$$={\left[\underset{z<0}{\inf }\exp [\Lambda (z)]\right]}^N$$

$$=\exp [-r_{ij}(\beta )N]$$

$$\exp [-r_{ij}N]$$

$$⟺N$$

$$A_v=\{\begin{array}{ll}1& \text{w.p. }{t}_{ij}^i,\text{ i.e., when candidate i approved but j not approved}\\ 0& \text{w.p. }1-t_{ij}^i-t_{ij}^j,\text{ i.e., when both approved, or neither approved}\\ -1& \text{w.p. }{t}_{ij}^j,\text{ i.e., when candidate j approved but i not approved}\end{array}$$

$$r_{ij}$$

$$\Lambda (z)$$

$$=\log \left[t_{ij}^i\exp (z)+t_{ij}^j\exp (-z)+(1-t_{ij}^i-t_{ij}^j)\right]$$

$$r_{ij}$$

$$=-\log \left[2\sqrt{t_{ij}^i{t}_{ij}^j}+1-t_{ij}^i-t_{ij}^j\right]$$

$$Q^N$$

$$\le \sum _{i\in C_s^{\ast },j\in C_t^{\ast },s<t}\exp [-r_{ij}N]$$

Full text

We provide a theoretical framework to make these choices. Each $K$-Approval or $K$-partial ranking mechanism (with a corresponding positional scoring rule) induces a learning rate for the speed at which the election recovers the asymptotic outcome. Given the voter choice distribution, the election planner can thus identify the rate optimal mechanism. Earlier work in this area provides coarse order-of-magnitude guaranties which are not sufficient to make such choices. Our framework further resolves questions of when randomizing between multiple mechanisms may improve learning for arbitrary voter noise models.

Finally, we use data from 5 large participatory budgeting elections that we organized across several US cities, along with other ranking data, to demonstrate the utility of our methods. In particular, we find that historically such elections have set $K$ too low and that picking the right mechanism can be the difference between identifying the ultimate winner with only a $80\%$ probability or a $99.9\%$ probability after 400 voters.

1 Introduction

Elections and opinion polls with many candidates and multiple winners are common. In participatory budgeting (PB), for example, people directly determine a part of the government’s budget Alós-Ferrer and Granić (2012); Goel et al. (2016). These elections often contain many candidate projects (up to 70, cf. Gelauff et al. (2018)) and only a few thousand voters, with potentially millions of dollars on the line Public Agenda (2016). Similarly, polls may compare tens of candidates and yet only sample hundreds of voters.

Unfortunately, the number of voters required to recover the asymptotic ranking or set of winners often scales, potentially exponentially, with the number of candidates Caragiannis and Micha (2017). Thus with many candidates, it is essential to use a voting mechanism that most efficiently elicits information from each voter.

In this work, we analyze positional scoring rules de Borda (1781); Young (1975), mechanisms in which each position in each voter’s personal ranking maps to a score given to the candidate that occupies that position. We focus on the special cases of such rules implied by $K$-Approval elicitation, in which each voter is asked to select their favorite $K$ candidates, as they the most commonly used such mechanisms in practice. Section 3 formalizes our model. Then:

Section 4.

For a given election, we show how the particular scoring rule used affects the rate at which the final outcome (asymptotic in the number of voters) is learned. These rates, based on large deviation bounds, extend and tighten the results of Caragiannis and Micha (2017), and are precise enough to determine, for example, which of $3$-Approval and $4$-Approval is better in a particular context. We focus on the goals of learning both a ranking over all candidates and identifying a subset of winners.

Section 5.1.

Leveraging these rates, we study when randomization between scoring rules can improve learning, extending previous results to general positional scoring rules, the goal of selecting a set of winners, and arbitrary noise models. In particular, we find that randomizing between scoring rules can never speed up learning, for arbitrary noise models. This contrasts to the case when one is restricted to $K$-Approval mechanisms.

Section 5.2.

For the Mallows model, we study how the optimal $K$ in $K$-Approval scales with the noise parameter, the number of candidates, and the number of winners desired. We find that, in contrast to design choices made in practice, one should potentially ask voters to identify their favorite half of candidates, even if the goal is to identify a single winner.

Section 6.

We apply our approach to experimental ballots attached to real participatory budgeting elections across several US cities, as well as other ranking data from a host of domains. We find that the exact mechanism used matters: in one setting, for example, asking voters to identify their favorite candidate results in only a $80\%$ chance of identifying the best candidate after 400 voters, while asking voters for their favorite $2$ candidates identifies the same best candidate $99.9\%$ of the time. Extending our theoretical insights, we find that, historically across elections, $K$ has been set too low for effective learning. We further identify real-world examples in which randomization would have sped up learning.

Our work bridges a gap between coarse theoretical analyses of voting rules and the fine-grained design questions a practitioner wishes to answer. Proofs are in the Appendix.

2 Related work

Our work is part of several strands of research on mechanisms that elicit peoples’ preferences. Aggregating voter rankings has a long history de Borda (1781); marquis de Condorcet (1785); Copeland (1951); Kemeny (1959); Young (1988).

Learning properties of voting rules. Most related are works that study the learning properties of voting rules, assuming that a “true” ranking. One approach is to specify a noise model under which voter preferences are drawn (e.g., Mallows, Plackett-Luce) and then derive error rates by the number of voters for maximum likelihood or similar estimators under the model Maystre and Grossglauser (2015); Zhao, Piech, and Xia (2016); Lu and Boutilier (2011); Guiver and Snelson (2009); Procaccia and Shah (2015); de Weerdt, Gerding, and Stein (2016); Chierichetti and Kleinberg (2014).

Caragiannis, Procaccia, and Shah (2013) ask similar questions to us: under what voter noise models do certain voting rules asymptotically recover the true underlying ranking, and how quickly do they do so. They define a class of voting rules and voter noise models under which a “true” ranking of candidates is eventually recovered. They further show that for a subset of this class (that does not contain positional scoring rules) and under the Mallows model, only a number of voters that is logarithmic in the number of candidates is required, where each voter provides a full ranking. Lee et al. (2014) develop an algorithm that can approximate the Borda rule, given a number of comparisons by each voter that is logarithmic in the number of candidates.

Most similar is that of Caragiannis and Micha (2017). They show that under the Mallows model, $K$-Approval with any fixed $K$ takes exponentially many voters (in the number of candidates) to recover the underlying ranking; on the other hand, $K$-approval with $K$ chosen uniformly at random for each voter takes only a polynomial number of voters.

These works provide order estimates for the learning rate, asymptotic in the number of candidates; fine-grained differentiation between different rules or $K$-Approval mechanisms for a given election is not possible. We provide the latter and show that it matters.

Other approaches to comparing mechanisms. Many works take an axiomatic and computational approach, comparing mechanisms that may produce different outcomes even given asymptotically many votes Fishburn and Gehrlein (1976); Fishburn (1978); Staring (1986); Tataru and Merlin (1997); Wiseman (2000); Ratliff (2003); Elkind et al. (2017); Aziz et al. (2015); Caragiannis et al. (2017); Aziz et al. (2017); Lackner and Skowron (2018a, b); Faliszewski and Talmon (2018). Caragiannis et al. (2019) for example show how to find a scoring rule that most agrees with a given partial ground truth ranking. In contrast, we compare mechanisms’ learning rates under a condition (formalized in Section 3.2) in which they produce the same asymptotic outcome.

Benade et al. (2018) and Gelauff et al. (2018) experimentally compare different mechanisms across several dimensions, including ease of use and consistency with another mechanism; the latter leverages data from a participatory budgeting election at a university.

Large deviation analysis of elicitation mechanisms. Theoretically, we leverage large deviation rates and Chernoff bounds to derive how quickly a given scoring rule learns its outcome; see work of Dembo and Zeitouni (2010) for an introduction to large deviations. This work is thus conceptually similar to work on elicitation design for rating systems  Garg and Johari (2018, 2019). In those works, the authors derive large deviation-based learning rates that depend on the questions that are asked to buyers as they review an item, where the goal is to accurately rank items; they further run an experiment on an online labor platform. In that setting, however, buyers rate a single item, and mechanisms are distinct based on the behavior they induce; in this work, voters see all the candidates and provide a partial ordering, and different designs (e.g., $3$-Approval vs $4$-Approval) constrain the types of orderings voters can provide.

3 Model

We now present our model and a condition under which different positional scoring rules induce the same asymptotic outcome.

3.1 Model primitives

We begin with the model primitives: candidates and voters, the election goal, and elicitation and aggregation.

Candidates and Voters. There is a set of $M$ candidates $C=\{1,\dots,M\}$, typically indexed by $i,j\in C$. There are $N$ voters $V=\{1,\dots,N\}$. Each voter $v\in V$ has a strict ranking of candidates $\sigma_{v}$, drawn independently and identically from probability mass function over strict rankings $F(\sigma)$. Let $i\succ_{\sigma}j$ denote that $i$ is preferred over $j$ in $\sigma$, and $\sigma(i)=k$ denote that candidate $i$ is in the $k$th position in $\sigma$.

A special case for $F$ is the Mallows model Mallows (1957), in which there is a “true” societal preference $\sigma^{*}$ from which each voter’s ranking is a noisy sample. In particular,

[TABLE]

Where $d(\sigma,\sigma^{*})$ is the Kendall’s $\tau$ distance between rankings $\sigma,\sigma^{*}$, and $\phi\in[0,1]$ is the noise parameter: the smaller it is, the more concentrated $F$ is around $\sigma^{*}$.

Election goal. We assume that the goal $G$ is to divide the candidates into $T$ disjoint, ordered tiers $G=\{C_{1},\dots,C_{T}\}$, such that $C=\cup_{t=1}^{T}C_{t}$, where candidate $i\in C_{s}$ is deemed societally preferable over $j\in C_{t}$ if $s<t$. The size of each tier is fixed before the election. For example, recovering a strict ranking over all candidates corresponds to $G=\{C_{1},\dots,C_{M}\}$, where $|C_{t}|=1$. Alternatively, identifying a set of $W$ winners, without distinguishing amongst the winners, corresponds to $G=\{C_{1},C_{2}\}$, with $|C_{1}|=W$.

In the main text and especially the empirics, we will focus on the task of selecting $W$ winners as it is the most common task in practice. However, this general notation allows comparison of the learning properties of different settings, and for example ask how much more expensive is it (in terms of the number of voters needed) to identify a strict ranking as opposed to just a set of winners.

Elicitation and Aggregation. Voters vote using an elicitation mechanism. Their votes are then aggregated using a positional scoring rule, parameterized as $\beta:\{1,\dots,M\}\mapsto\mathbb{R}$. We consider the following mechanisms:

$K$-Ranking

Voter $v$ ranks her favorite $K$ candidates, i.e., reveals $\{(i,{\sigma_{v}}(i))\,:\,{\sigma_{v}}(i)\leq K\}$. Candidate $i$ then receives a score $s_{iv}=\beta({\sigma_{v}}(i))$ if ranked, [math] otherwise. For example, $\beta(k)=M-k$ for the Borda count.111In Borda, candidates not ranked receive a score $(M-K-1)/2$, consistent with assuming they are all tied in position $(K+1)$.

$K$-Approval

Voter $v$ selects her favorite $K$ candidates, i.e., reveals $\{i\,:\,{\sigma_{v}}(i)\leq K\}$. A candidate receives a score $s_{iv}=1$ for being selected, [math] otherwise.

$\beta$ encodes both elicitation and aggregation. For example, $K$-Approval is equivalent to $K$-ranking with score function $\beta(k)=\mathbb{I}[k\leq K]$. Furthermore, note that given $K$-ranking data, one can simulate $K^{\prime}$-ranking elicitation for $K^{\prime}\leq K$ with a $\beta$ s.t. $\beta(k)=0$ for $k>K^{\prime}$.

The scoring rule $\beta$ is a design choice made by the election organizer, and so we will refer to $\beta$ as the election’s design. We restrict ourselves to non-constant, non-increasing scoring rules, i.e., $\beta\in\mathcal{B}=\{\beta:\forall k<\ell\in{1,\dots,M},\beta(k)\geq\beta(\ell),\text{ and }\exists k<\ell,\beta(k)>\beta(\ell)\}$.

Outcome. After $N$ voters, candidate $i$’s cumulative score is $s_{i}^{N}=\frac{1}{N}\sum_{v=1}^{N}s_{iv}$. Candidates are ranked in descending order of score, to form ranking ${\sigma}^{N}$, with ties broken uniformly at random. We denote the outcome after $N$ voters, corresponding to the goal $G$, as $O^{N}(M,F,\beta,G)$. For example, for the goal of selecting $W$ winners, $O^{N}(M,F,\beta,G)$ is simply the top $W$ candidates in ${\sigma}^{N}$. When $(M,F,\beta,G)$ is clear from context, we will refer to the outcome as $O^{N}$.

As the number of voters $N\to\infty$, candidate scores $s_{i}^{N}\to\mathbb{E}_{F}[s_{iv}]\triangleq s_{i}$ by the law of large numbers; when such expected scores are distinct, i.e., $s_{i}\neq s_{j}$ for $i\neq j$, then $\sigma^{N}\to\sigma^{*}$ for some ranking $\sigma^{*}$. However, note that there may exist an asymptotic outcome $O^{N}\to O^{*}$ even without an asymptotic ranking $\sigma^{N}\to\sigma^{*}$, as long as expected scores $s_{i}$ and goal $G$ are such that candidates with identical expected scores are sorted into the same tier.

3.2 Asymptotic design invariance

The asymptotic outcome $O^{*}$ of an election may vary with the scoring rule $\beta$. For example, there may be a different winner if voters are asked to identify their favorite two candidates than if they identify their single favorite candidate, if the winner in the latter case is a polarizing candidate. As an axiomatic comparison between outcomes is out of the scope of this paper, we restrict our attention to cases where all “reasonable” choices of different $\beta$ asymptotically result in the same outcome (where “reasonable” corresponds to the set of scoring rules $\mathcal{B}$ defined above).

Definition 1**.**

A setting $(M,F)$ is asymptotically design-invariant for goal $G$ if any reasonable $\beta$ induces the same outcome asymptotically. $\exists O^{*}:\forall\beta\in\mathcal{B},$

[TABLE]

Such design invariance only occurs under a fairly strong condition on the voter preference distribution: that the candidates can be separated into tiers (according to goal $G$) such that candidates in higher tiers are strictly more likely to be ranked by a voter in the top $k$ positions, for all $k<M$, than are candidates in lower tiers.

Proposition 1**.**

A setting $(M,F)$ for goal $G$ is asymptotically design-invariant if and only if there exist candidate tiers $O^{*}=\{C_{1}^{*}\dots C_{T}^{*}\}$ (corresponding to $G$) s.t. $\forall s<t$: $i\in C^{*}_{s},j\in C^{*}_{t}\implies$ $\text{Pr}_{F}(\sigma_{v}(i)\leq k)>\text{Pr}_{F}(\sigma_{v}(j)\leq k)$, $\forall k\in\{1\dots M-1\}$.

Note that this condition is stronger than stochastic dominance as the inequality is strict for every position $k$.

This proposition connects to Caragiannis, Procaccia, and Shah (2013) as follows: they prove that many rules (including all positional scoring rules and the Bucklin rule) asymptotically recover the base ranking $\sigma^{*}$ of a generalization of the Mallows model in which the probability $F(\sigma)$ of a ranking $\sigma$ is monotonic in the distance $d(\sigma,\sigma^{*})$, where distance function $d$ is itself in some general class that contains the Kendall’s $\tau$ distance. Their results directly imply that such noise models, including the standard Mallows model, are asymptotically design-invariant for any goal $G$.

However, for goals $G$ where recovering a full ranking is unnecessary, the condition in Proposition 1 is weaker than the assumptions of Caragiannis, Procaccia, and Shah (2013); there need not even be a single base ranking $\sigma^{*}$. For example, when $G$ such that we wish to select a set of $W$ winners, $F$ corresponding to a mixture of Mallows models – with all possible permutations of the $W$ candidates in the top $W$ positions in the base rankings – would still be design-invariant. Constructing a general class of ranking noise models that satisfies this property is an avenue for future work.

Assuming asymptotic design-invariance on voter preferences $F$ may seem restrictive. However, absent axioms – that are precise enough for design purposes – to prefer one scoring rule $\beta$ over another, the assumption allows us to proceed in a principled manner. We believe it is unlikely that such precise, satisfactory axioms exist generally. In the Appendix, we provide a simple example (similar to that of Staring (1986)) where $1$-Approval and $2$-Approval select disjoint sets of $2$-Winners, and such examples can be adapted more generally to selecting $W$ winners from either $K$-Approvals or $K^{\prime}$-Approvals. In participatory budgeting with the goal of identifying 6-10 winning projects out of over twenty projects, it is unclear whether there is a principled reason to prefer $4$-Approval over $8$-Approval. However, such axioms would be an interesting avenue for future work.

Furthermore, in Section 6.2 we show that design invariance is often approximately satisfied in practice, especially for identifying a small set of winners, using data from a wide range of participatory budgeting and other elections.

4 Learning Rates and Optimal Design

Different elicitation and aggregation mechanisms may take different amounts of voters to learn the asymptotic outcome. For example, suppose we want to identify the worst candidate out of 100, where the voter’s rankings are drawn from a Mallows model with $\phi>0$. Then, asking each voter to identify their single favorite candidate will eventually identify the worst candidate, but after many more voters than if we ask each voter to identify their least favorite candidate. We make such learning rates precise in this section. Our results in this section extend those of Caragiannis and Micha (2017) as discussed above, both to arbitrary positional scoring rules and by providing tighter bounds for how a scoring rule affects the convergence rate. These rates are precise enough to design scoring rules, for example comparing $4$-Approval and $8$-Approval in the above example.

4.1 Learning rates

We begin by deriving rates for how quickly a given positional scoring rule $\beta$ learns its asymptotic outcome $O^{*}$ (given it exists), as a function of the voter preference model $F$. In particular, we use large deviation rates at which a scoring rule learns Dembo and Zeitouni (2010).

Definition 2**.**

Consider a sequence $\{A_{N}\geq 0\}_{N\in\mathbb{N}}$, where $A_{N}\to 0$ . Value $r>0$ is the large deviation rate for $A_{N}$ if

[TABLE]

When $r>0$ exists, $A_{N}\to 0$ exponentially fast, with exponent $r$ asymptotically, i.e., $A_{N}$ is $e^{-rN\pm o(N)}$. These rates provide us both upper and lower bounds for the probability of an error or the number of errors in an outcome after $N$ voters, up to polynomial factors. In particular, in the propositions below, we will calculate the large deviation rate of errors in the outcome. We will also then provide (loose) upper bounds for such errors after $N$ voters that hold without any missing polynomial factors, for any $N$. These upper bounds are equivalent to Chernoff bounds.

The particular forms for these rates, derived below for general noise models $F$, may seem complex. However, they are useful both for theoreticians and practitioners. For example, in Section 5.1, we use the structure of such rates to resolve open questions regarding when randomization between mechanisms can help learn the outcome from votes drawn from an arbitrary noise model. In Section 6, we show that learning rates – even when empirically calculated – reflect the true behavior of errors in real elections with a small number of voters; we then use empirically calculated learning rates to draw design insights across elections.

Rates for separating two candidates. We now derive the large deviation learning rates for recovering the true ordering between a pair of candidates $i,j$, given noise model $F$. These rates will directly translate to the learning rate for the overall election, given some goal $G$.

Proposition 2**.**

Fix scoring rule $\beta\in\mathcal{B}$, voter distribution $F$, and consider candidates $i,j$ such that $s_{i}>s_{j}$. Then, the probability of making a mistake in ranking these two candidates after $N$ voters, $\text{Pr}(\sigma^{N}(i)>\sigma^{N}(j))$, goes to zero with large deviation rate

[TABLE]

Further, the following upper bound holds for any $N$.

[TABLE]

The proof follows directly from writing a random variable for the event of making a mistake after $N$ voters and then applying known large deviation rates. This simplicity emerges because positional scoring rules are additive across voters.

The proposition establishes that – for a fixed number of candidates $M$ and voter noise model $F$ – the probability of making a mistake on any single pair of candidates $i,j$ decreases exponentially with the number of voters, at a rate governed by the scoring rule $\beta$ and the candidates’ relative probabilities of appearing at each position of a voter’s preference ranking. The rate $r_{ij}(\beta)$ is non-negative, and and larger values correspond to faster learning of the relative ranking of $i,j$. Note that for notational convenience, we suppress $F$ in the argument for the rate.

For general $\beta$, we cannot find a closed form for $r_{ij}(\beta)$. However, the structure of this rate, in particular that of the argument in the $\log(\cdot)$, will directly let us show that randomization cannot help learning outcomes among positional scoring rules, for arbitrary noise models $F$.

For $K$-Approval voting, further, the rate simplifies.

Proposition 3**.**

Consider $\beta$ consistent with $K$-Approval voting for some fixed $K$, and candidates $i,j$ such that $s_{i}>s_{j}$. Then the large deviation rate $r_{ij}(\beta)$ in Proposition 2 is

[TABLE]

Where $t^{i}_{ij}(K)\triangleq\text{Pr}_{F}(\sigma_{v}(i)\leq K,\sigma_{v}(j)>K)$, i.e., the probability that a voter approves $i$ but not $j$.

The proof follows directly from the structure of $\beta$ for $K$-Approval, $\beta(k)=\mathbb{I}[k\leq K]$; for each pair of candidates, the sufficient statistics are how often each candidate appears in a voter’s top $K$ list but the other candidate does not.

We overload notation and use $K$ directly in the argument for $r_{ij}(K)$. This rate function $r_{ij}(K)$ is convex in the probabilities $t^{i}_{ij}(K),t^{j}_{ij}(K)$; this fact will let us show that randomization, even among $K$-Approval mechanisms, cannot help learning the relationship of any pair of candidates.

Rates for learning the outcome. In general, the rates at which one learns each pair of candidates immediately translate to rates for learning the entire outcome $O^{*}$.

Proposition 4**.**

Consider goal $G$ and $\beta\in\mathcal{B}$ such that $O^{N}\to O^{*}$. Let $Q^{N}$ be the expected number of errors in the outcome after $N$ voters, $\sum_{i\in C_{s}^{*},j\in C_{t}^{*},s<t}\text{Pr}(\sigma^{N}(i)>\sigma^{N}(j))$. Then $Q^{N}$ goes to zero with large deviation rate

[TABLE]

Further, the following upper bound holds for any $N$.

[TABLE]

The large deviation rate $r(\beta)$ thus provides a tight characterization for how many voters it takes to (with high confidence) recover the asymptotic outcome of an election. Note that the goal plays an important role: for selecting $W$ winners, for example, it is not important to learn the exact relationship among candidates $\{1,\dots,W\}$, speeding up outcome learning. Design $\beta$ also matters; e.g., even amongst approval voting mechanisms, $K=1$ vs $K=5$ will produce substantially different $t_{ij}^{i}(K)$. To derive learning rates for $K$-Approval for any given noise model or using real-world data, one simply needs to calculate these values. We do so numerically for the Mallows model and empirically with real world data in Sections 5.2 and 6, respectively.

4.2 Optimal design and discussion

Now that we can quantify how quickly a given scoring rule $\beta$ learns its asymptotic outcome, we apply our framework to designing elections, i.e., choosing an optimal scoring rule $\beta$. For the rest of this work, we assume that the setting $(M,F)$ is asymptotically design-invariant for the goal $G$, i.e., there exists an outcome that is asymptotically induced by every reasonable scoring rule. Then, the design of an election $\beta$ only affects the rate at which the election converges to the asymptotic outcome $O^{*}$, as calculated above. With no other constraints, then, the design challenge is simple: find the rate optimal $\beta$.

Definition 3**.**

A scoring rule $\beta^{*}\in\mathcal{B}$ is rate optimal if it maximizes the rate in Proposition 4. $K^{*}$-Approval is Approval rate optimal if it maximizes the rate among $K$-Approval mechanisms.

Rate optimal designs $\beta$ learn the outcome faster than others in the number of voters, and so are preferable to other designs. What influences how quickly a design $\beta$ learns?

$\mathbb{E}_{F}\left[\exp\left(z\left(\beta(\sigma_{v}(i))-\beta(\sigma_{v}(j))\right)\right)\right]$ must be small (near zero) for negative $z$, and so $\beta(k)-\beta(k^{\prime})$ must be large when $\text{Pr}(\sigma_{v}(i)=k,\sigma_{v}(j)=k^{\prime})$ is large. In other words, a scoring rule must reward a candidate achieving a position in a voter’s ranking that is only achieved by asymptotically high-ranking candidates. For example, if it is common for worse candidates to be ranked second in a given voter’s ranking but not to be ranked first, then $\beta(1)\gg\beta(2)$ would be beneficial.

Note that finding such designs requires knowledge of the voter noise model $F$, which in many settings may not be available before the election. However, next in Sections 5 and 6, we show that there are valuable insights that apply across elections, including how our approach has informed participatory budgeting deployments.

5 Theoretical Design Insights

The learning rates derived in the previous section provide election design insights, even before our approach is applied to real-world data. In particular, in this section, we first extend the previous literature on the (potential) benefits of randomizing between mechanisms. Then, we study the task of selecting $W$ winners using $K$-Approval voting.

5.1 When does randomization help?

We now consider the question of whether randomizing between mechanisms in an election may speed up learning. By randomization, we mean: consider a set of scoring rules $B=\{\beta_{1},\dots,\beta_{P}\}\subseteq\mathcal{B}$; elicitation and aggregation for a given voter is done according to a scoring rule picked at random from $B$, where $\beta_{p}$ is selected with probability $d_{p}$.

Note that the learning rate of such randomized schemes can be calculated as before, by summing across $\beta_{p}$ inside the $\mathbb{E}[\cdot]$ of $r_{ij}(\beta)$ or – for $B$ consisting only of $K$-Approval votes – directly through the resulting probability that the voter approves $i$ but not $j$. We use $r_{ij}(B,D),r(B,D)$ to denote the candidate pairwise and overall outcome learning rates, respectively, for randomized mechanism $(B,D)$, where $B=\{\beta_{1},\dots,\beta_{P}\}\subseteq\mathcal{B}$ and $D=\{d_{1},\dots,d_{P}\}$.

It is known that in some settings randomization improves learning, asymptotically in the number of candidates. Caragiannis and Micha (2017) provide an example in which randomizing uniformly between all possible $K$-Approval mechanisms outperforms any static $K$-Approval elicitation, when the goal is to rank all the candidates. Their insight is that, under the Mallows model and under a fixed $K$, either the first two candidates will be hard to distinguish from each other, or the last two will, and randomizing between mechanisms balances learning each pair.

We now study randomization for the goal of selecting $W$ winners and for arbitrary positional scoring rules and voter noise models. Our first result is that randomizing between scoring rules does not help, for any voter noise model, in contrast to the case when restricted to approval votes.

Theorem 1**.**

Randomization does not improve the outcome learning rate for any asymptotically design-invariant noise model $F$ or goal $G$. For any randomized scoring rule mechanism $(B,D)$, where $B\subset\mathcal{B}$, for any $F$, $G$, the scoring rule $\beta^{*}(k)=\sum_{p}d_{p}\beta_{p}(k)$ satisfies $r(\beta^{*})\geq r(B,D)$.

The result follows from the fact that $\mathbb{E}_{F}\left[\exp\left(z\left(\beta(\sigma_{v}(i))-\beta(\sigma_{v}(j))\right)\right)\right]$ is convex in $\beta(k)$, for all $i,j,z,F$. Then, given a randomization over $\beta_{1},\dots\beta_{P}$, we can increase $-\inf_{z}\log(\cdot)$ by decreasing its argument, by instead using the static scoring rule defined by the corresponding convex combination of $\beta_{1},\dots\beta_{P}$. Note that such a negative result cannot be obtained via analysis that is asymptotic in the number of candidates; we need learning rates for a given election.

Next, we further refine the result of Caragiannis and Micha (2017), by showing that the “pivotal pair” feature of their example – where different pairs of candidates dominate the learning rate for different mechanisms – is key. In particular, our next result establishes, again for any noise model, that randomization amongst $K$-Approval mechanisms cannot help separate any given pair of candidates.

Theorem 2**.**

Randomization amongst $K$-Approval mechanisms does not improve the learning rate for separating a given pair of candidates $i,j$ for any asymptotically design-invariant noise model $F$ or goal $G$. For any randomized $K$-Approval mechanism $(B,D)$, where $\beta_{p}\in B$ corresponds to $p$-Approval, for any $F$, $G$, there exists a mechanism $K_{ij}^{*}$-Approval such that $r_{ij}(K_{ij}^{*})\geq r_{ij}(B,D)$.

The proof relies on the pairwise rate function $r_{ij}(K)$ being convex in the approval probabilities $t^{i}_{ij}(K),t^{j}_{ij}(K)$.

This theorem directly implies that, for the Mallows model, randomization among $K$-Approval voting cannot speed up learning when the goal is to identify a set of $W$ winners, as opposed to when the goal is to rank.

Corollary 1**.**

Randomization among $K$-Approval mechanisms does not improve the learning rate for selecting $W$ winners from the Mallows model. For any randomized $K$-Approval mechanism $(B,D)$, where $\beta_{p}\in B$ corresponds to $p$-Approval, for selecting $W$ winners from the Mallows model, there exists an Approval rate optimal mechanism $K^{*}$-Approval such that $r(K^{*})\geq r(B,D)$.

The proof simply notes that under the Mallows model with this goal, the candidate pair $W,W+1$ (when candidates are indexed according to reference distribution $\sigma^{*}$) is pivotal regardless of the $K$-Approval mechanism used. This corollary does not extend to arbitrary noise models, where randomization amongst $K$-approval mechanisms may improve the learning rate.

Theorem 3**.**

Randomization among $K$-Approval mechanisms may improve the learning rate for the goal of selecting $W$ winners. There exist asymptotically design-invariant settings $(M,F)$ for the goal of selecting $W$ winners such that a randomized $K$-Approval mechanism $(B,D)$, where $\beta_{p}\in B$ corresponds to $p$-Approval, satisfies

[TABLE]

We prove the result two ways: (1) we construct an example in which candidate $h$ is asymptotically selected, and candidates $i,j$ are not. Which of $h\succ i$ or $h\succ j$ is the pivotal pair (determines the overall rate function) depends on the $K$-Approval mechanism used, and randomizing between two mechanisms improves the overall rate; (2) perhaps more interestingly, we find many examples in our real PB elections and other ranking data in which randomization would have sped up learning for the task of selecting a set of winning candidates (see Section 6.4).

5.2 $K$-Approval for selecting $W$ winners

One of the most common voting settings is identifying a set of $W$ winners using $K$-Approval, whether in representative democracy elections (typically $K=W=1$), polling for such elections (where the goal often is to identify the top few candidates out of many, especially in primary races), or crowd-sourcing labels (where one wants one or a few labels for an item out of many possible ones). Here, we study how to design such elections, i.e., how to choose the best $K$, i.e., the one that maximizes the learning rate. For simplicity, we work with the Mallows model, extending the resulting insights to real-world data in the next section.

Recall that in a Mallows model, each voter’s ranking is a noisy sample from a reference distribution $\sigma^{*}$. With this symmetric model, one may believe that setting $K=W$ is always optimal. For example, when noise parameter $\phi=0$ and so each voter’s ranking is exactly $\sigma^{*}$, $K=W$ is optimal; in fact, any other design $K\neq W$ fails to correctly identify the set of winners even asymptotically: it would not distinguish among the first $K$ candidates in $\sigma^{*}$ or among the last $M-K$ candidates. However, our next result establishes that the cases with $\phi>0$ are different.

Theorem 4**.**

Under the Mallows model and the goal of selecting $W$ winners, $W$-Approval may not be Approval rate optimal.

We prove the theorem by example. To find this example and to generate the plots discussed next, we use an efficient dynamic program to exactly calculate the joint distributions of the locations $\sigma_{v}(i),\sigma_{v}(j)$ of pairs of candidates $i,j$ in a voter’s ranking, given the Mallows noise parameter; we can then directly calculate $t_{ij}^{i}(K),t_{ij}^{j}(K)$ and thus the learning rate for each $K$-Approval mechanism. This program leverages Mallows repeated insertion probabilities Lu and Boutilier (2014); Diaconis (1988) and may be of independent interest for numerical analyses of the Mallows model.

Numerical analysis. We now numerically analyze, for the Mallows model, how the Approval rate approval $K$-Approval mechanism varies with the Mallows noise parameter $\phi$, the number of candidates $M$, and the number of winners $W$. Recall that the Mallows model is asymptotically design invariant, so different mechanisms only differ in how quickly they learn the asymptotic outcome.

In Figure 1(a), the goal is to select $W=1$ winner, and $\phi$ and $M$ are varied. With low noise, $\phi\lessapprox.5$, it is rate optimal to use $1$-Approval, i.e., ask each voter to select their favorite candidate, regardless of how many candidates there are. However, with higher noise $\phi$, as the number of candidates in the election increases, so does the $K$ in the optimal $K$-Approval mechanism. For $\phi=.999,M=50$, for example, it is best to ask each voter to select their favorite $25$ candidates, even if the task is to identify the single best candidate according to the reference distribution $\sigma^{*}$.

Similarly, Figure 1(b) shows how the rate optimal $K$-Approval mechanism changes with the number of winners desired and the noise parameter, fixing the number of candidates at $M=50$. Again with high noise, it is best to ask voters to identify their favorite half of candidates, regardless of how many winners need to be identified. With low noise, however, $W$-Approval is optimal to select $W$ winners.

Overall, the analysis suggests that with higher noise in the voter model, one should tend toward asking voters to rank their favorite half of candidates, regardless of $M$ and $W$.

The high-noise setting may seem unrealistic; however, as we will see in the next section, which $K$-Approval mechanism is rate optimal in practice often scales like the high noise settings, consistent with the idea that voting distributions in practice do not look like they are drawn from a low-noise Mallows model. We now turn to such empirical analyses.

6 Empirics and PB deployments

We now apply our insights to practice. We focus on $K$-Approval voting, as opposed to general scoring rules. This section is organized as follows. In Section 6.1, we describe our data sources. We validate our model in Section 6.2; first, we demonstrate that large deviation rates effectively describe how quickly various mechanisms learn; next, we show that in practice voter noise models are approximately design invariant. In Section 6.3, we show that the insights from Section 5.2 regarding optimal approval mechanisms extend to practice. Finally in Section 6.4 we note that we find many examples in practice where randomizing between $K$-Approval mechanisms improves learning.

6.1 Data description

We leverage two data sources (detailed dataset information is in Appendix Table 1). First, we have partnered with dozens of local governments to help run participatory budgeting (PB) elections in the last five years. These elections have used a variety of methods, primarily $K$-Approval; our data in this work comes from 5 elections where $K$-Ranking was used, including 3 recent elections where $K=10$. This data is particularly useful as PB is among the most common types of elections with many candidates and several winners, with several theoretical analyses Goel et al. (2016); Garg et al. (2019); Freeman et al. (2019).

Second, we use data available on PrefLib Mattei and Walsh (2013); O’Neill (2013); Regenwetter et al. (2007, 2008); Popov, Popova, and Regenwetter (2014), limiting ourselves to 28 elections with at least 5 candidates and 700 voters who provided full rankings. This ranking data spans many domains, from people’s sushi preferences to Glasgow City Council elections. This domain breadth supports the broad applicability of the design insights explored in this section.

We focus on ranking data to be able to simulate counter-factuals for the same election: with $K$-Ranking data, we can simulate what would have occurred with any $K^{\prime}$-Approval elicitation mechanism, for $K^{\prime}\leq K$ (assuming no behavioral quirks). With approval data, on the other hand, one cannot compare the mechanism to any other for that given election.

One challenge is that ranking many candidates is onerous, and so voters rank at most 14 candidates in our dataset. For the data we use from on PrefLib, full rankings (rankings up to the number of candidates) are available. In the PB elections in our partner cities, typically each voter ranks or selects her favorite $K\ll M$ candidates.

6.2 Model validation

Our model and design approach has two components that must be validated: (1) that learning rates can effectively be used to compare different mechanisms, and (2) that design invariance (approximately) holds in practice.

Large deviation rates as effective proxies for learning. We now confirm that, for a given election, empirically calculated large deviation learning rates are effective proxies for the rate at which the error in recovering the asymptotic output decreases as the number of voters increases (even though large deviation learning rates are only asymptotically valid in the number of voters). As examples, we first identify three elections and goals for which many of the potential $K$-Approval mechanisms return exactly the same asymptotic outcome. Then, we bootstrap voters from the available data of voters and empirically calculate the errors made in identifying the winning set of candidates. We further calculate the large deviation learning rates for these mechanisms, using $F$ implied by the voting data and the formula in Proposition 4.222Given an empirical $\hat{F}$, learning rates can be numerically calculated: the $\inf_{z}[\cdot]$ is a convex minimization problem.

Figure 2(a) shows the resulting errors over time for one such election where $4$-Rankings are available. We further plot $e^{-rN}$ for each mechanism, i.e., the error over time implied by the learning rate (up to polynomial factors). This plot, along with Appendix Figure 3, yields several insights:

1. (1)

The mechanism matters: when selecting 1 winner from the election in Figure 2(a) after $400$ votes, there is $20\%$ chance of not picking the ultimate winner if $1$-Approval is used. With $2$ or $3$-Approval, this number is $0.1\%$. The winner appears often in a voter’s top two or three positions (but not necessarily first), while the ultimate second place candidate often falls outside the top three. Scoring rules that reward top three placements thus perform well. 2. (2)

The learning rates effectively capture the behavior of the empirical error: both comparatively across mechanisms, as well as the asymptotic rate (slope of the line in log scale). This property enables use of large deviation learning rates as proxies for learning even in elections with a small number of voters. 3. (3)

Ranking $K$ candidates rather than selecting $K$ candidates is more onerous for voters. However, it does not always provide more information in terms of learning rates, as in the examples in Appendix Figure 3.

Design invariance in practice. Design invariance does not strictly hold in any election in our dataset (as expected as the condition is strong). However, it approximately holds. Similar mechanisms produce the same asymptotic outcome for many tasks. Figure 2(b) shows, for example, the average overlap across elections in the top $4$ candidates identified by each mechanism. (Appendix Figure 4 shows the same plot for the top $1$ and $3$ candidates, as well as the average Kendall’s $\tau$ rank correlation between the full rankings identified by different mechanisms). Furthermore, we find many elections and goals where most mechanisms return the same asymptotic answer, as in the elections we leverage for the plots showing learning rates are effective proxies. This relative consistency, especially for similar mechanisms, enables us to compare different mechanisms by their learning rates.

6.3 $K$-Approval for selecting $W$ winners

In Section 5.2, we showed for the Mallows model how the rate optimal $K$-Approval mechanism changes with the noise parameter $\phi$, the number of candidates, and the number of winners. We now show this scaling in practice.

For every election in our dataset, we find the Approval rate optimal mechanism (among $K$ we can simulate) for every goal of selecting $W$ winners, for $1\leq W\leq M$. We then run a regression across all the elections for which $K$ is rate optimal, versus the number of winners desired and the number of candidates; see Table 2 in the Appendix for the regression table. While there is some variation across elections, the number of candidates and winners proves a reasonable metric across elections for the rate approval $K$-Approval mechanism ($R^{2}\approx.27$).

The regression confirms the idea that in practice, one should regularize toward asking voters to choose their favorite half the candidates. For picking a small subset of winners $W\approx 4$ out of more than $10$ candidates, for example, one should ask voters to provide their favorite $K\approx 6$ candidates, with $K>W$. This suggestion directly counters common practice. In the PB elections that we have helped run, for example, $4$ or $5$-Approval is most typical, even though ultimately 6-10 projects may be funded (out of $\approx 15$-$20$).

Then, in Figure 5 in the Appendix, we plot the line induced from the regression coefficients with the Mallows rate optimal lines, for $M\leq 10$ candidates. Comparing to the rate optimal mechanisms for the Mallows model with various $\phi$ (within the candidate range for which we have empirical data), we find that empirical data behaves most closely to a Mallows model with noise parameter $\phi\in[.8,.9]$. (We are not claiming that empirical data is drawn from a Mallows model; it most certainly is not, with factors such as polarizing projects important in practice). This coarse comparison provides an approximate expected scaling behavior for elections with many candidates.

6.4 Randomization in practice

We find 16 examples in which randomizing between two $K$-Approval mechanisms leads to faster learning than using either mechanism separately, including 8 examples where such randomization beats the Approval rate optimal mechanism. Table 3 in the Appendix contains details.

7 Discussion

We show that in elections with many candidates, the elicitation mechanism and corresponding scoring rule used affect how quickly the final outcome is learned. The learning speed differential between mechanisms can be the difference between identifying the ultimate winner with only a $80\%$ probability or a $99.9\%$ probability after 400 voters, for example. We then provide design decisions that emerge when our framework is applied to data from real elections. When using $K$-Approval to select a small number of $W$ winners, for example, it is often better to ask voters to identify their favorite $K>W$ candidates. The insights from this work should be applicable in a variety of such settings, from elections to crowdsourcing labeling tasks.

There are several important, open research avenues. Most importantly, in real elections maximizing the rate at which the final outcome is identified is not the only goal, and future work should seek to balance such multiple objectives.

For example, there may be axiomatic reasons to prefer one elicitation mechanism over another, e.g., that the final outcome corresponds to the candidate(s) that the most voters indicate is their first choice. Another objective may be to minimize the cognitive load imposed on voters. Asking voters to provide a full ranking over the candidates and then using a rate-optimal scoring rule trivially provides faster learning than any other mechanism. However, asking voters to rank 20 candidates is prohibitive in many settings. Future empirical work, in line with that of Benade et al. (2018) and Gelauff et al. (2018), should study the cognitive load various mechanisms impose on voters, to better understand the trade-off between the objectives.

Acknowledgments

We thank our Participatory Budgeting city partners, especially those in Boston, Durham, and Rochester. We also thank anonymous reviewers for their comments. This work was supported in part by the Stanford Cyber Initiative, the Office of Naval Research grant N00014-19-1-2268, and National Science Foundation grants 1544548 and 1637397.

Appendix A Empirics additional information

Appendix B Proofs

B.1 Asymptotic design-invariance

See 1

Proof.

[TABLE]

$\implies$. By the definition of asymptotically design-invariant,

[TABLE]

For this $O^{*}=\{C_{1}^{*},\dots,C_{T}^{*}\}$, we show by contradiction that $\forall s<t$: $i\in C^{*}_{s},j\in C^{*}_{t}\implies$ $\text{Pr}_{F}(\sigma_{v}(i)\leq k)>\text{Pr}_{F}(\sigma_{v}(j)\leq k)$, $\forall k\in\{1\dots M-1\}$: Suppose $\exists i\in C^{*}_{s},j\in C^{*}_{t},s<t,k\in\{1\dots M-1\}$ such that $\text{Pr}_{F}(\sigma_{v}(i)\leq k)\leq\text{Pr}_{F}(\sigma_{v}(j)\leq k)$. Then, let

[TABLE]

Then, $\mathbb{E}[s_{iv}]=\beta(M)+\beta(k)\text{Pr}_{F}(\sigma_{v}(i)\leq k)\leq\mathbb{E}[s_{jv}]$. Then, with positive probability,

[TABLE]

.

$\impliedby$. Suppose there exists such a $O^{*}$. Then, $\forall\beta\in\mathcal{B}=\{\beta:\forall k<\ell\in{1\dots M},\beta(k)\geq\beta(\ell),\text{ and }\exists k<\ell,\beta(k)>\beta(\ell)\}$: Suppose $i\in C_{s}^{*},j\in C_{t}^{*},s<t$:

[TABLE]

Where the strict inequality follows as $\exists m:\beta(m)-\beta(m+1)>0$. Then, for all candidates $i\in C_{s}^{*},j\in C_{t}^{*},s<t$, by the strong law of large numbers $\lim_{N\to\infty}s_{i}^{N}>\lim_{N\to\infty}s_{j}^{N}$ w.p. $1$. Thus, $\lim_{N\to\infty}\mathcal{O}(M,N,F,\beta,G)=O^{*}$ w.p. $1$. ∎

Remark 1**.**

The following example, with candidates $A,B,C,D$ leads to a disjoint set of $2$ winners with $1$-Approval and $2$ approval, respectively

With $1$-Approval, candidates $A,D$ are selected. With $2$-Approval, $B,C$ are selected.

B.2 Learning rates

Notation.

$t_{ij}^{i}(K)$ is the probability that $i$ is approved but $j$ is not, using $K$-Approval.

For convenience, we overload the rate function $r(\cdot)$:

• •

$r_{ij}(\beta)$ is as defined in Proposition 2, the large deviation rate to learn a pair of candidates $i,j$ given scoring rule $\beta$, for a fixed $F$ that should be clear from context. When a goal $G$ is clear from context, $r(\beta)$ is as defined in Proposition 4, the minimum over $r_{ij}(\beta)$ for candidate pairs that are in different asymptotic tiers.

• •

$r_{ij}(K)$ is as defined in Proposition 3, the large deviation rate to learn a pair of candidates $i,j$ using $K$-Approval. $r(K)$ is analogous to the previous item when using $K$ approval.

• •

$r(a,b)$ is the large deviation rate to learn a pair of candidates $i,j$ using approval voting when the probability that $i$ is approved but $j$ is not is $a$; and $b$ is the probability that $j$ is approved but $i$ is not is not.

When which rate function we mean is clear from context, we may drop the argument $(\cdot)$ and just write $r_{ij}$ or $r$.

Remark 2**.**

$r(a,b)>r(c,d)$* when $a>c,b\leq d$, OR $a\geq c,b<d$.*

Proof.

$\gamma(a,b)=\sqrt{ab}+1-a-b$ is strictly concave in $a,b$, with maximum at $a=b$. Thus, holding either $a$ or $b$ constant and moving the other farther away strictly decreases $\gamma$, and thus strictly increases $r$. ∎

See 2

Proof.

Define the following random variable for each voter $v\sim F$:

[TABLE]

Then, $\sigma^{N}(i)<\sigma^{N}(j)$ when $A^{N}=\sum_{v=1}^{N}A_{v}>0$, and $\mathbb{E}[A_{v}]>0$ by supposition. Let

[TABLE]

Then, by basic large deviation bounds (see, e.g. Dembo and Zeitouni (2010)):

[TABLE]

And, applying Chernoff bounds, we get the standard relationship to the large deviation rate, giving an upper bound for the probability of error directly, including any polynomial factors out front:

[TABLE]

Then, $\text{Pr}(\sigma^{N}(i)<\sigma^{N}(j))>1-\epsilon$ when

[TABLE]

∎

See 3

Proof.

With $K$-approval voting, $A^{ij}_{v}$ becomes

[TABLE]

Then,

[TABLE]

The $\inf(\Lambda)$ is attained at $z=\frac{1}{2}\log\frac{t_{ij}^{j}}{t_{ij}^{i}}$ ($\Lambda$ is convex in $z$, and so setting the first derivative to zero finds the $\inf$). And so

[TABLE]

∎

See 4

Proof.

By the Union bound

[TABLE]

Now, using large deviation properties:

By supposition, $Q^{N}\to 0$, and so $-Q^{N}$ approaches [math] from below. Then,

[TABLE]

Line (1) follows from: $\forall a^{\epsilon}_{i}\geq 0$, ${\lim\sup}_{\epsilon\to 0}\left[\epsilon\log\left(\sum_{i}^{N}a^{\epsilon}_{i}\right)\right]=\max^{N}_{i}{\lim\sup}_{\epsilon\to 0}\epsilon\log(a^{\epsilon}_{i})$. See, e.g., Lemma 1.2.15 in Dembo and Zeitouni (2010) for a proof of this property.

Thus $r$ is the large deviation rate for $Q^{N}$.

∎

B.3 Design insights

See 1

Proof.

From Proposition 2, for a given scoring rule $\beta$ and pair of candidates $i,j$, the learning rate is

[TABLE]

Similarly, if we use scoring rules $\{\beta^{u}\}_{u=1}^{P}$, each with probability $d^{u}$, then,

[TABLE]

Now, for a single scoring rule $\beta(\cdot)$, let

[TABLE]

Below, we show that $\gamma(\beta(1),\dots,\beta(M))$ is convex in $\beta(k),\forall k,z$. Then, by convexity, $\forall z$

[TABLE]

and so

[TABLE]

The left hand side is equal to the argument inside the $-\log(\cdot)$ for the rate function for randomizing between scoring rules $\{\beta^{u}\}_{u=1}^{P}$, each with probability $d^{u}$, and the right hand side is the argument inside for the rate function for instead using the single scoring rule $\beta^{*}$ defined as the convex combination of $\{\beta^{u}\}_{u=1}^{P}$. Then, as $-\log(x)$ is decreasing in $x$, we have that

[TABLE]

.

As this holds for each pair of candidates $i,j$ simultaneously, we are done.

Proof that $\gamma(\beta(1),\dots,\beta(M))\triangleq\mathbb{E}_{F}\left[\exp\left[z\left[\beta(\sigma_{v}(i))-\beta(\sigma_{v}(j))\right]\right]\right]$ is convex in $\beta(k),\forall k$.

We directly calculate the Hessian of $\gamma$ and note that it is diagonally dominant and thus positive semidefinite. For notational convenience, we let $\beta_{k}=\beta(k)$, and $\sigma(k,\ell)=\text{Pr}_{F}(\sigma_{v}(i)=k,\sigma_{v}(j)=\ell)$. Of course, $\sigma(k,k)=0$, as we assume each voter has a strict ranking as her preference.

[TABLE]

Thus, the Hessian of $\gamma$ is diagonally dominant with non-negative diagonal elements: $\forall k$,

[TABLE]

and so the Hessian is positive semi-definite. Thus, $\gamma$ is convex in $\beta_{k}$.

∎

See 2

Proof.

From Proposition 3, for $k$-Approval,

[TABLE]

Where $t^{i}_{ij}$ is the probability that $i$ is approved but $j$ is not.

This rate function is convex in $t^{i}_{ij},t^{j}_{ij}$:

• •

$r_{ij}(a,b)=h(g(a,b))$, where $h(x)=-\log(x)$, $g(a,b)=2\sqrt{ab}+1-a-b$.

• •

$g(a,b)$ is concave in $a,b$

• •

$h$ is convex, and $\tilde{h}$ is non-increasing, where $\tilde{h}(x)=\begin{cases}h(x)&x>0\\ \infty&x\leq 0\end{cases}$ is the extended value function of $h$.

• •

By convex composition rules, $r_{ij}(a,b)$ is convex (see, e.g., page 84 of Boyd and Vandenberghe (2004)).

The result follows by convexity. Consider a randomization of $K$-Approval mechanisms for $K\in\{1,\dots,M-1\}$, where $K$-Approval is used with probability $d^{K}$.

The resulting approval probabilities are: $t^{i}_{ij}=\sum_{K=1}^{M-1}d^{K}t^{i}_{ij}(K),t^{j}_{ij}=\sum_{K=1}^{M-1}d^{K}t^{j}_{ij}(K)$. By convexity:

[TABLE]

We note that, unlike the previous proof, we cannot conclude in general that randomization cannot improve the rates at which the outcome is learned (in fact, Theorem 3 establishes otherwise). That is because while the same $\beta^{*}$ could be said to be rate optimal (compared to the randomized mechanism) for every pair of candidates simultaneously in that proof, in this proof $\arg\max_{K}r_{ij}(K)$ may change based on the pair $i,j$.

∎

See 1

Proof.

When selecting $W$ winners out of $M$ candidates, we need to separate candidates $1\dots W$ from candidates $W+1\dots M$. It is easy to show that the pivotal pair, regardless of which $K$ is used in $K$-Approval, is $W$,$W+1$. Applying Theorem 2, then, randomization cannot help the overall rate. ∎

See 4

Proof.

We prove the result by providing an example where it is not optimal. Suppose there are 4 candidates, and we wish to select $3$ winners, i.e., separate the first three items from the last item.

Let the items in the reference ranking be, in order, $i=1,2,3,4$, respectively.

A Mallows model (with parameter $\phi=\frac{p}{1-p}$, where $p$ is the probability of flipping a given pair of candidates) can be sampled by repeated insertion (Lu and Boutilier, 2014; Diaconis, 1988): starting from the first item in the reference ranking, there exists probability $\tilde{p}_{ij}=\frac{\phi^{i-j}}{1+\phi^{1}+\dots+\phi^{i-1}}$ at which item $i$ can be inserted into position $j\leq i$, independently of how items above it were inserted, such that the resulting ranking distribution matches the Mallows model.

Using this repeated insertion property for our example, we can derive $p_{\ell k}$, the probability at which item $3$ is in position $\ell$ and item $4$ is in position $k$ after sampling from a Mallows model with parameter $\phi$.

In particular, if $\ell<k$, $p_{\ell k}$ is exactly the probability that item $3$ is inserted in position $\ell$ and item $4$ is inserted in position $k$. If $k>\ell$, however, it is the probability that item $3$ is inserted in position $\ell-1$ and then pushed down when item $4$ is inserted in position $k$. (More generally, it turns out, the exact probability for an item appearing in a given position in the Mallows model can be calculated using a simple dynamic program, a fact that does not appear to be documented elsewhere but may be independently useful. We used this dynamic program to find this given example).

Then, for our example

[TABLE]

Recall that $t_{ij}^{i}(K)$ is the probability that $i$ is approved but $j$ is not, using $K$-Approval. Then, if we use $3$-approval and $2$-approval, respectively:

[TABLE]

Then, recall the rate between items $i,j$ using $K$ approval is

[TABLE]

When there is low noise, e.g., $\phi=.1$ ($p=.091$):

[TABLE]

But when there is high noise, e.g., $\phi=.8$ ($p=.44$):

[TABLE]

Note that the same example works for selecting 1 winner out of the 4 candidates, as the repeated insertion model can be run in reverse.

∎

See 3

Proof.

We provide two proofs: a numeric example from a real-world election, and a contrived, constructed example.

Numeric example found in a real election. In Durham Ward 1, to select 2 winners, randomizing between 3 and 4-Approval is better than either individually, even though asymptotically the mechanisms pick the same set of winners. The critical pair with $2$-Approval is with the candidate asymptotically ranked 1st, and the best item not selected. With $3$-Approval, it is with the candidate asymptotically ranked 2nd, and the same best item not selected.

We will call these items $h$, $i,j$ (the one not selected) respectively. The respective probabilities of being selected alone:

[TABLE]

And the resulting rates (using the formula in Proposition 3) are:

[TABLE]

Thus randomization improves learning.

Constructed example with design invariance. We now construct a fully design-invariant example with the same flavor as the numeric example, where which pair is critical changes with the mechanism.

Consider three candidates $h,i,j$, such that $h$ is asymptotically in the set of $W$ winners and $i,j$ are not. Thus, we need the rates at which $h$ is separated from both $i,j$. Let $W=K<L=K+1$.

We prove the result by giving an example where: it is easier to separate $h$ from $i$ using $K$-Approval, and easier to separate $h$ from $j$ using $L$-Approval. Using $K$-Approval, $r_{hj}$ asymptotically dominates the rate at which the overall outcome is learned, and using $L$-Approval, $r_{hi}$ does. Further, randomizing between the two mechanisms improves the two rates that dominate enough such that the overall rate is improved.

We need to show the following hold for our example: one of the rates between candidates $h$ and $i,j$ are smaller than other rates, i.e., dominate the overall learning rate when $K$ and/or $L$ approval is used; randomization between $K$ and $L$ approval helps the minimum rate between candidates $h$ and $i,j$; $K^{\prime}$-Approval ($K^{\prime}\neq K,K^{\prime}\neq L$) produces a worse rate than either $K$ or $L$ approval; and this example is asymptotically design-invariant.

We prove each of these conditions in turn after specifying the example.

Recall that $t_{ij}^{i}(k)$ is the probability that $i$ is approved but $j$ is not, using $k$-Approval. Here, we will use:

[TABLE]

. The end row labeled “Total value” then sums up these values.

Example Specification. Consider $F$ such according to the following table, where the first column is the probabilities of the positions in the second set of columns. The third set of columns indicates whether those set of positions contribute to the given probabilities, for easy accounting.

The table does not specify the probabilities of other candidates appearing in any position, so it is possible that they dominate the learning rate (are hardest to learn). (In particular, if the same, asymptotically non-winning candidate $q$ is always in position $L$ in the case in row $3$, then it may be hard to separate it from candidate $h$ using $L$ approval). However, we can specify the example further to ensure this does not happen.

Suppose candidates are indexed by their order in some strict ranking $\sigma^{*}$. Then, candidates $h=K,i=L=K+1,j=K+2$. Further suppose that candidates in $\{1\dots K-1\}$ always occupy, in order except in case of row 10, the best positions in a voter’s ranking that are not reserved for candidates $h,i,j$ in the table above.

For candidates $q\in\{K+3\dots M\}$, we have to be more careful to avoid the case in parenthesis above. Suppose these $Q=M-K+2$ candidates fill up the bottom spots in a voter’s ranking in a uniform at random order. In other words, they occupy spots $L+3\dots M$, and the worst spot among whichever of $K,L,L+1,L+2$ is missing in each row in the table above.

Rates between the $h$ and $i,j$ dominate the overall learning rate using $K$ or $L$ approval.

We are now ready to show the first claim that learning between candidates $h$ and $i,j$ is hardest (when using either $K$ or $L$ approval).

By the specification above, candidates in $\{1\dots K-2\}$ are always approved, and so learning between those candidates and any non-winning candidate is faster than any large deviations rate. Similarly, candidate $K-1$ always is ranked higher than candidates $q\in\{K+3\dots M\}$, and it is approved alone with high enough probability.

Then, the other candidates who may dominate the learning rate are candidate $K-1$ (in separation from $i,j$), or $q\in\{K+3\dots M\}$ (in separation from $h$). From the above table:

[TABLE]

Now, suppose $3a>\frac{T_{1}}{Q}$ and $Q>2$. (Both conditions occur for $Q$ large enough). Then, applying Remark 2 regarding learning rates being larger when the arguments are farther away from one another (holding one fixed), the resulting rates with these candidates are dominated by (larger than) the rates between candidates $h$ and $i,j$, discussed next.

Randomizing improves the minimum rate between candidates $h$ and $i,j$. By Remark 2,

[TABLE]

Using $K$-Approval:

[TABLE]

Using $L$-Approval:

[TABLE]

Randomizing – For any $0<p<1$, eliciting $K$-Approval with probability $p$, and $L$-Approval otherwise:

[TABLE]

$K^{\prime}$-Approval ($K^{\prime}\neq K,K^{\prime}\neq L$) produces a worse rate than either $K$ or $L$ approval.

For any $K^{\prime}<K-1$, $h$ is approved with probability $\epsilon_{2}<\epsilon$, and $i,j$ are never approved. Then, the rate between $h$ and $i,j$ is $-\log(1-\epsilon_{2})\to 0$ as $\epsilon\to 0$. Identically, for $K^{\prime}\geq L+2=K+3$, both $h$ and $i,j$ are approved except with some probability $\epsilon_{2}<\epsilon$.

For $K^{\prime}=K-1$, $h$ is approved without $i$ with probability $a+\epsilon_{2}$ (for some $\epsilon_{2}<\epsilon$), and $i$ is approved without $h$ with probability $a$. Then, the rate between $h$ and $i$ is $-\log(2\sqrt{(a+\epsilon_{2})a}+1-2a-\epsilon_{2})\to 0$ as $\epsilon\to 0$.

For $K^{\prime}=K+2=L+1$, $h$ is approved without $i$ with probability $T_{2}-a+\epsilon_{2}$, and $i$ is approved without $h$ with probability [math]. Then, the rate between them is $-\log(1-T_{2}+a-\epsilon_{2})$. For $T_{2}$ small enough, this is a worse rate than using $K$ or $L$ approval.

The example described is asymptotically design-invariant. From the above table, the probability that candidate $c\in\{1\dots M\}$ is in position $k$ or better, i.e., $\sigma(c)\leq k$ is:

Conditions on constants in problem. For the above claims to hold, we set conditions on the constants in the problem. They are

[TABLE]

This is a feasible set of constraints: $Q$ can be set large enough to meet conditions 3,4,5 for any fixed $T_{1},a,T_{2}$ that meet condition 1. Condition 2 is weaker than the last condition. That leaves the last condition along with the first one.

[TABLE]

which holds for $T_{1}$ large enough, and $T_{2},\epsilon$ small enough.

∎