Teaching decision theory proof strategies using a crowdsourcing problem

TL;DR

This paper introduces a new example involving crowdsourcing in cosmology to teach decision theory proof strategies, specifically minimax and Bayes approaches, aimed at first-year graduate students.

Contribution

It provides a clear, practical example illustrating standard decision rule derivation techniques, enhancing teaching methods for decision theory.

Findings

Demonstrates how to derive minimax and Bayes decision rules using the example

Highlights advantages and disadvantages of Bayesian and minimax approaches

Applicable to teaching decision theory in graduate courses

Abstract

Teaching how to derive minimax decision rules can be challenging because of the lack of examples that are simple enough to be used in the classroom. Motivated by this challenge, we provide a new example that illustrates the use of standard techniques in the derivation of optimal decision rules under the Bayes and minimax approaches. We discuss how to predict the value of an unknown quantity, $\theta \! \in \! \{0,1\}$, given the opinions of $n$ experts. An important example of such crowdsourcing problem occurs in modern cosmology, where $\theta$ indicates whether a given galaxy is merging or not, and $Y_1, \ldots, Y_n$ are the opinions from $n$ astronomers regarding $\theta$. We use the obtained prediction rules to discuss advantages and disadvantages of the Bayes and minimax approaches to decision theory. The material presented here is intended to be taught to first-year graduate students.

Tables

Table 1. Table 1: An example of crowdsourcing in cosmology: three astronomers give their opinion about weather a given galaxy is a merger or not. What is the optimal way to combine such opinions in order to recover the truth?

	Is the galaxy a merger?
Galaxy	Expert 1	Expert 2	Expert 3
	No	Yes	Yes
	No	Yes	No
	Yes	Yes	No

Table 2. Table 2: Bayes decision rules that are obtained in Example 1 for each of the priors that are considered. 0 / 1 0 1 0/1 is used when both decisions are Bayes (i.e., they both have the same expected posterior loss).

	Bayes decision rule using
Experts’ opinions	Prior 1	Prior 2	Prior 3
(0,0,0)	0	0	0
(1,0,0)	1	0/1	1
(0,1,0)	0	0/1	1
(0,0,1)	0	0	0
(1,1,0)	1	1	1
(1,0,1)	1	0/1	0
(0,1,1)	0	0/1	0
(1,1,1)	1	1	1

Equations

$${\delta }_c^{\ast }(\mathbf{y},u)$$

$$\{\begin{array}{ll}{\delta }^{\ast }(\mathbf{y},u)=1& \text{, if }\frac{{\mathbb{E}}_P[L(0,\theta )|\mathbf{y}]}{{\mathbb{E}}_P[L(1,\theta )|\mathbf{y}]}>1\\ {\delta }^{\ast }(\mathbf{y},u)=0& \text{, if }\frac{{\mathbb{E}}_P[L(0,\theta )|\mathbf{y}]}{{\mathbb{E}}_P[L(1,\theta )|\mathbf{y}]}<1\end{array}$$

$${\delta }^{\ast }(\mathbf{y},u)=\{\begin{array}{ll}1& \text{, if }\frac{{\mathbb{E}}_P[L(0,\theta )|\mathbf{y}]}{{\mathbb{E}}_P[L(1,\theta )|\mathbf{y}]}>1\\ 0& \text{, if }\frac{{\mathbb{E}}_P[L(0,\theta )|\mathbf{y}]}{{\mathbb{E}}_P[L(1,\theta )|\mathbf{y}]}<1\\ u& \text{, if }\frac{{\mathbb{E}}_P[L(0,\theta )|\mathbf{y}]}{{\mathbb{E}}_P[L(1,\theta )|\mathbf{y}]}=1\end{array}$$

$$\{\begin{array}{ll}\frac{{\mathbb{E}}_P[L(0,\theta )|\mathbf{y}]}{{\mathbb{E}}_P[L(1,\theta )|\mathbf{y}]}>1& ⟺\frac{\mathbb{P}(\theta =1|\mathbf{y})}{\mathbb{P}(\theta =0|\mathbf{y})}>1⟺\mathbb{P}(\theta =1|\mathbf{y})>\frac{1}{2}\\ \frac{{\mathbb{E}}_P[L(0,\theta )|\mathbf{y}]}{{\mathbb{E}}_P[L(1,\theta )|\mathbf{y}]}<1& ⟺\frac{\mathbb{P}(\theta =1|\mathbf{y})}{\mathbb{P}(\theta =0|\mathbf{y})}<1⟺\mathbb{P}(\theta =1|\mathbf{y})<\frac{1}{2}\end{array}$$

$$\mathbb{P}(\theta =1|\mathbf{y})>\frac{1}{2}$$

$$⟺2\sum _{i=1}^n\log \left(\frac{{\gamma }_i}{1-{\gamma }_i}\right)y_i>\log \left(\frac{1-c}{c}\right)+\sum _{i=1}^n\log \left(\frac{{\gamma }_i}{1-{\gamma }_i}\right).$$

$$\mathbb{P}(\theta =1|\mathbf{y})>\frac{1}{2}$$

$$\mathbb{P}(\theta =1|\mathbf{y})<\frac{1}{2}$$

$${\delta }_M(\mathbf{y},u)=1⟺\bar{\mathbf{y}}>\frac{1}{2}$$

$$\displaystyle\begin{cases}R(\delta^{*}_{c},0)&={\mathbb{P}}\left(\sum_{i=1}^{n}w_{i}^{*}Y_{i}>C^{*}\!\>\Big{\lvert}\>\!\theta=0\right)+\frac{1}{2}{\mathbb{P}}\left(\sum_{i=1}^{n}w_{i}^{*}Y_{i}=C^{*}\!\>\Big{\lvert}\>\!\theta=0\right)\\ R(\delta^{*}_{c},1)&={\mathbb{P}}\left(\sum_{i=1}^{n}w_{i}^{*}Y_{i}<C^{*}\!\>\Big{\lvert}\>\!\theta=1\right)+\frac{1}{2}{\mathbb{P}}\left(\sum_{i=1}^{n}w_{i}^{*}Y_{i}=C^{*}\!\>\Big{\lvert}\>\!\theta=1\right)\end{cases}$$

$$\displaystyle{\mathbb{P}}\left(\sum_{i}w_{i}^{*}Y_{i}>C^{*}\!\>\Big{\lvert}\>\!\theta=0\right)+\frac{1}{2}{\mathbb{P}}\left(\sum_{i=1}^{n}w_{i}^{*}Y_{i}=C^{*}\!\>\Big{\lvert}\>\!\theta=0\right)$$

$$=$$

$$\displaystyle{\mathbb{P}}\left(\sum_{i=1}^{n}w_{i}^{*}Y_{i}>C^{*}\!\>\Big{\lvert}\>\!\theta=0\right)+\frac{1}{2}{\mathbb{P}}\left(\sum_{i=1}^{n}w_{i}^{*}Y_{i}=C^{*}\!\>\Big{\lvert}\>\!\theta=0\right)$$

$$=$$

$$\displaystyle{\mathbb{P}}\left(\sum_{i=1}^{n}w_{i}^{*}Y_{i}<C^{*}\!\>\Big{\lvert}\>\!\theta=1\right)+\frac{1}{2}{\mathbb{P}}\left(\sum_{i=1}^{n}w_{i}^{*}Y_{i}=C^{*}\!\>\Big{\lvert}\>\!\theta=1\right)$$

$$=$$

$$E_{P^{\ast }}[R({\delta }^{\ast },\theta )]\ge E_P[R(\delta ,\theta )]$$

$$\Theta =\{(\theta ,{\gamma }_1,\dots ,{\gamma }_n):\text{mbox}\theta \in \{0,1\},{\gamma }_i\in [0,1]\text{mbox}\forall i=1,\dots ,n\}$$

$${\delta }_c^{\ast \ast }(\mathbf{y},u)$$

$$\mathbb{P}(\theta =1|\mathbf{y})$$

$$=\frac{\mathbb{P}(\theta =1)}{\mathbb{P}(\mathbf{Y}=\mathbf{y})}\int \mathbb{P}(\mathbf{Y}=\mathbf{y}|\theta =1,\mathbf{\gamma })f(\mathbf{\gamma })d\mathbf{\gamma }$$

$$=\frac{c}{\mathbb{P}(\mathbf{Y}=\mathbf{y})}\int \prod _{i=1}^n({\gamma }_i{)}^{y_i}(1-{\gamma }_i{)}^{1-y_i}\prod _{i=1}^n\frac{\Gamma ({\alpha }_i+{\beta }_i)}{\Gamma ({\alpha }_i)\Gamma ({\beta }_i)}({\gamma }_i{)}^{{\alpha }_i-1}(1-{\gamma }_i{)}^{{\beta }_i-1}d\mathbf{\gamma }$$

$$=\frac{c}{\mathbb{P}(\mathbf{Y}=\mathbf{y})}\prod _{i=1}^n\frac{\Gamma ({\alpha }_i+{\beta }_i)}{\Gamma ({\alpha }_i)\Gamma ({\beta }_i)}\int ({\gamma }_i{)}^{{\alpha }_i+y_i-1}(1-{\gamma }_i{)}^{{\beta }_i-y_i}d\mathbf{\gamma }$$

$$=\frac{c}{\mathbb{P}(\mathbf{Y}=\mathbf{y})}\prod _{i=1}^n\frac{\Gamma ({\alpha }_i+{\beta }_i)}{\Gamma ({\alpha }_i)\Gamma ({\beta }_i)}\frac{\Gamma ({\alpha }_i+y_i)\Gamma ({\beta }_i+1-y_i)}{\Gamma ({\alpha }_i+{\beta }_i+1)}$$

$$\mathbb{P}(\theta =0|\mathbf{y})$$

$$\frac{\mathbb{P}(\theta =1|\mathbf{y})}{\mathbb{P}(\theta =0|\mathbf{y})}>1$$

$$⟺\sum _{i=1}^n\log \left(\frac{\Gamma ({\alpha }_i+y_i)\Gamma ({\beta }_i+1-y_i)}{\Gamma ({\alpha }_i+1-y_i)\Gamma ({\beta }_i+y_i)}\right)>\log \left(\frac{1-c}{c}\right)$$

$$\frac{\mathbb{P}(\theta =1|\mathbf{y})}{\mathbb{P}(\theta =0|\mathbf{y})}>1$$

$$\frac{\mathbb{P}(\theta =1|\mathbf{y})}{\mathbb{P}(\theta =0|\mathbf{y})}<1$$

$${\delta }^{cf}(\mathbf{y},u)=u$$

$$\{\begin{array}{ll}\mathbb{P}(\theta =1|\mathbf{y})& =\frac{1}{2\mathbb{P}(\mathbf{Y}=\mathbf{y})}{\prod }_{i=1}^n\frac{\Gamma (1+y_i)\Gamma (2-y_i)}{2}\\ \mathbb{P}(\theta =0|\mathbf{y})& =\frac{1}{2\mathbb{P}(\mathbf{Y}=\mathbf{y})}{\prod }_{i=1}^n\frac{\Gamma (2-y_i)\Gamma (1+y_i)}{2}\end{array}$$

$$\{\begin{array}{l}R({\delta }^{cf},(0,\mathbf{\gamma }))=\mathbb{P}({\delta }^{cf}(\mathbf{Y},U)=1|(0,\mathbf{\gamma }))=\mathbb{P}(U=1||(0,\mathbf{\gamma }))=\frac{1}{2}\\ R({\delta }^{cf},(1,\mathbf{\gamma }))=\mathbb{P}({\delta }^{cf}(\mathbf{Y},U)=0|(1,\mathbf{\gamma }))=\mathbb{P}(U=0||(1,\mathbf{\gamma }))=\frac{1}{2}\end{array}$$

$${\Theta }^{\ast }=\left\{(\theta ,{\gamma }_1,\dots ,{\gamma }_n):\text{mbox}\theta \in \{0,1\},{\gamma }_i\ge \frac{1}{2}+ϵ\text{mbox}\forall i=1,\dots ,n\right\}$$

$$\underset{\delta }{\inf }\underset{\xi \in {\Theta }^{\ast }}{\sup }\text{mbox}R(\delta ,\xi )\le \underset{\xi \in {\Theta }^{\ast }}{\sup }\text{mbox}R({\delta }_M,\xi )$$

Full text

Keywords: Bayes decision, minimax decision, crowdsourcing problem

AMS Classification: 6201, 62C10, 62C20

Running title: Decision theory using a crowdsourcing problem

1 Introduction

Decision theory formally compares statistical methods. It is deeply linked to most Bayesian procedures (Casella and Berger, 2002; DeGroot, 2005; Parmigiani and Inoue, 2009). Also, in the minimax approach, decision theory is the standard criterion for defining frequentist optimality of nonparametric methods (Wasserman, 2006; Tsybakov, 2008). However, there are few published examples in which both the minimax and Bayes solutions involve elementary calculations, which tends to obscure teaching this topic. Here, we explore a motivating example where calculations are simple and yet illustrate standard techniques for finding optimal estimators. We also use this example to discuss advantages and disadvantages of the Bayes and minimax approaches to decision theory. The material presented here is intended to be used for first-year graduate level students.

The main element of the motivating example is a sequence, ${\mathbf{Y}}=\{Y_{1},\ldots,Y_{n}\}\!\in\!\{0,1\}^{n}$, composed of the opinions of $n$ experts about an unknown quantity, $\theta\in\{0,1\}$. For example, $\theta$ might be 1 if a given galaxy is merging with another galaxy and [math] otherwise. It is an important question in cosmology to infer the value of $\theta$, because the number of merger galaxies plays a key role in testing theories about the evolution of the Universe (Lotz et al., 2004; Freeman et al., 2013). In this case, $Y_{1},\ldots,Y_{n}$ would be the opinions from $n$ astronomers regarding the value of $\theta$, obtained using an image of that galaxy; see Table 1 for some examples from the Cosmic Assembly Near-IR Deep Extragalactic Legacy Survey (CANDELS; Grogin et al. 2011). Given the low resolution of the images, it is common for astronomers to disagree regarding the value of $\theta$. Hence, we are interested in using all the information in the opinions, $y_{1},\ldots,y_{n}$, to infer the value of $\theta$. This example is a building block for crowdsourcing models, such as the one developed in Izbicki and Stern (2013b), which are becoming increasingly popular due to platforms such as Amazon Mechanical Turk and projects such as Galaxy Zoo111https://www.galaxyzoo.org/ and foldit222http://fold.it.

As a minor element of the motivating example, we also allow the use of a fair coin flip, $U$, to infer about $\theta$. That is, $U\sim\text{Bernoulli}(0.5)$ and $U$ is independent of ${\mathbf{Y}}$. The coin flip can be useful in making a decision when one is indifferent about the options that are available. The importance of this element is discussed in more detail in subsection 2.2.

The motivating example can be framed as a decision problem, hereafter called the crowdsourcing problem. For each set of expert opinions and coin flip outcome, the decision-maker must make a prediction, $a\in\{0,1\}$, about the value of $\theta$. Given values for $a$ and $\theta$, the decision-maker is penalized according to $L(\theta,a)={\mathbb{I}}(\theta\neq a)$, that is, errors are penalized with a loss of one unit and correct decisions have zero loss. The goal of the decision-maker is to design a decision rule $\delta\!:\!\{0,1\}^{n}\times\{0,1\}\longrightarrow\{0,1\}$ which, for each collection of expert opinions ${\mathbf{y}}:=(y_{1},\ldots,y_{n})$ and coin flip outcome, $u$, associates a decision $\delta({\mathbf{y}},u)\in\{0,1\}$. The decision-maker wants a decision rule that leads to a small loss. More precisely, the risk (Wasserman, 2006)[p.228] of the decision rule $\delta$, $R(\delta,\theta)={\mathbb{E}}_{{\mathbf{Y}}\!\>\lvert\>\!\theta}[L(\delta({\mathbf{Y}},U),\theta)]$, should be small. Because the risk depends on the unknown value of $\theta$ and it is typically not possible to find a decision rule that minimizes the risk for every $\theta$, the following optimality criteria are typically used:

• •

[Minimax] $\delta^{*}$ is a minimax rule if $\sup_{\theta}R(\delta^{*},\theta)=\inf_{\delta}\sup_{\theta}R(\delta,\theta)$

• •

[Bayes] $\delta^{*}$ is a Bayes decision rule if ${\mathbb{E}}_{P}[R(\delta^{*},\theta)]=\inf_{\delta}{\mathbb{E}}_{P}[R(\delta,\theta)]$

${\mathbb{E}}_{P}[R(\delta,\theta)]$ is called the Bayes risk of $\delta$. The expectation that appears in the Bayes risk is with respect to $P$, the prior distribution on $\theta$.

In order to explore the crowdsourcing problem, we require some assumptions:

Assumption 1

$Y_{1},\ldots,Y_{n}$* are independent (conditionally on $\theta$).* 2. 2.

$\gamma_{i}:={\mathbb{P}}(Y_{i}=1\!\>\lvert\>\!\theta=1)={\mathbb{P}}(Y_{i}=0\!\>\lvert\>\!\theta=0)$. 3. 3.

$n$* is odd.*

Observe that $\gamma_{i}$ is the accuracy of expert $i$. According to Assumption 1.2, for each expert, the chance of making a mistake does not depend on the value of $\theta$. Assumption 1.3 simplifies some derivations by considering that one cannot obtain ties between experts’ 0’s and 1’s.

Section 2 analyzes the case in which $\gamma_{i}$’s are known in advance. Section 3 analyzes the case in which the $\gamma_{i}$’s are unknown. In pursuance of simplifying the text, these sections use the following notation:

Notation 1

$c:=P(\theta=1)$. 2. 2.

$\bar{{\mathbf{Y}}}:=\frac{1}{n}\sum_{i=1}^{n}Y_{i}$* and $\bar{{\mathbf{y}}}:=\frac{1}{n}\sum_{i=1}^{n}y_{i}$.*

2 Experts’ accuracies are known

2.1 Bayes decision rule

This subsection uses the crowdsourcing problem with known expert accuracies to illustrate a strategy that is commonly used to find Bayes decision rules. In order to simplify calculations and make the Bayes decision rules more interpretable, we use the following notation:

Notation 2

$w_{i}^{*}:=2\log\left(\frac{\gamma_{i}}{1-\gamma_{i}}\right)$* and $C^{*}:=\log\left(\frac{1-c}{c}\right)+\sum_{i=1}^{n}\frac{w_{i}^{*}}{2}$.*

Observe that $w_{i}^{*}$ increases monotonically with the $i$-th expert’s accuracy, $\gamma_{i}$. Furthermore, if $w_{i}^{*}<0$, then $\gamma_{i}<\frac{1}{2}$, that is, the $i$-th expert is less accurate than a coin flip. Similarly, if $w_{i}^{*}>0$, then the $i$-th expert is more accurate than a coin flip. Next, we derive Bayes rules in the crowdsourcing problem.

Theorem 1

In the crowdsourcing problem with known expert accuracies, for each $c\in[0,1]$, the following decision rule is a Bayes decision rule

[TABLE]

The decision rule given by Theorem 1 admits an intuitive interpretation. If a weighted sum of the opinions of the experts is greater than the cutoff, $C^{*}$, then one decides that $\theta$ is 1. $C^{*}$ varies according to the prior probability assigned to $\theta$. Also, the more an expert is accurate, the larger the weight assigned to him. Observe that, if expert $i$ is worse than a coin flip ($w_{i}^{*}<0$), then the model favors $\theta=1-y_{i}$, the opposite of the expert’s opinion. Roughly speaking, the decision will be determined by the most accurate experts.

In order to prove Theorem 1, a commonly used technique is given by Theorem 2:

Theorem 2** ((Parmigiani and Inoue, 2009))**

A decision rule $\delta^{*}$ is a Bayes decision rule if and only if, for each ${\mathbf{y}}$ and $u$, $\delta^{*}({\mathbf{y}},u)$ minimizes ${\mathbb{E}}_{P}[L(\delta({\mathbf{y}},u),\theta)|{\mathbf{y}}]$.

Theorem 2 allows one to find a Bayes decision rule by determining a decision for one value of ${\mathbf{y}}$ at a time. This procedure is called the extensive form of a decision problem. It is generally easier to solve a problem in the extensive form than to directly prove that a given decision rule has a Bayes risk that is smaller than every other decision rule (the latter approach is called the normal form of a decision problem). The following paragraphs show how the solution of the extensive form for the crowdsourcing problem can be simplified to Theorem 1.

In the crowdsourcing problem $\delta(\textbf{y},u)\in\{0,1\}$. Using this observation, one can obtain from Theorem 2 the following lemma:

Lemma 1

$\delta^{*}$* is a Bayes decision rule in the crowdsourcing problem if and only if, for all ${\mathbf{y}}$ and $u$,*

[TABLE]

One can conclude from Lemma 1 that there may exist several Bayes decision rules for the crowdsourcing problem. If there exists some y such that $\frac{{\mathbb{E}}_{P}[L(0,\theta)\!\>\lvert\>\!{\mathbf{y}}]}{{\mathbb{E}}_{P}[L(1,\theta)\!\>\lvert\>\!{\mathbf{y}}]}=1$, then one is indifferent between choosing either [math] or $1$ when observing y. In other words, in this case the choice that is made is not relevant for the goal of obtaining a Bayes decision rule. Corollary 1 presents a particular Bayes decision rule that aligns with the one in Theorem 1. This case of Bayes decision rule is emphasized because it is used in subsection 2.2.

Corollary 1

Let $\delta^{*}$ be defined as

[TABLE]

$\delta^{*}$* is a Bayes decision rule for the crowdsourcing problem.*

Also observe that $L(0,0)=L(1,1)=0$ and $L(0,1)=L(1,0)=1$. Therefore, one can obtain Lemma 2

Lemma 2

[TABLE]

Furthermore,

[TABLE]

Plugging Notation 2 into Equation 2.1, one obtains that

[TABLE]

Similarly,

[TABLE]

Combining the above derivation with Corollary 1 and Lemma 2, Theorem 1 is obtained.

Besides the importance of Bayes decision rules from the Bayesian perspective, these decision rules can also have good frequentist properties. This is exemplified in the next subsection, which illustrates how to find a minimax decision rule through Bayes decision rules. This proof strategy is commonly used.

2.2 Minimax decision rule

This subsection exemplifies a strategy that is often used to find minimax decision rules. In particular, it proves the following theorem in the crowdsourcing problem:

Theorem 3

$\delta^{*}_{\frac{1}{2}}$* (theorem 1) has constant risk and is minimax.*

Observe that, if ${\mathbb{P}}(\theta=1)=c=\frac{1}{2}$, then one is indifferent between $\theta=1$ and $\theta=0$ before observing the experts’ opinions. This is a possible intuition for why it is plausible that the Bayes decision rule $\delta^{*}_{\frac{1}{2}}$ is also minimax. Furthermore, Corollary 2 shows how $\delta^{*}_{\frac{1}{2}}$ can be simplified into a familiar decision rule.

Corollary 2

If $\gamma_{1}=\ldots=\gamma_{n}>\frac{1}{2}$, that is, all experts have the same accuracy and the accuracy is better than that of a coin flip, then the decision rule $\delta_{M}$ defined as

[TABLE]

is equal to the Bayes decision rule $\delta^{*}_{\frac{1}{2}}({\mathbf{y}},u)$ and is a minimax decision rule. Observe that $\bar{{\mathbf{y}}}=\frac{1}{2}$ cannot occur because of Assumption 1.3.

The decision rule $\delta_{M}$ is known as the majority rule and has been used in crowdsourcing problems (Raykar et al., 2010). Such an agreement between $\delta^{*}_{\frac{1}{2}}$ and $\delta_{M}$ might also improve the plausibility of $\delta^{*}_{\frac{1}{2}}$ being minimax.

In order to prove that $\delta^{*}_{\frac{1}{2}}$ is indeed minimax, we use a standard technique, that is described in Theorem 4.

Theorem 4** ((Wasserman, 2006))**

If $\delta^{*}$ is a Bayes decision rule against some prior distribution and the risk of $\delta^{*}$, $R(\delta^{*},\theta)$, is constant on $\theta$, then $\delta^{*}$ is a minimax decision rule.

Theorem 4 provides a proof technique for finding minimax rules that is usually better than directly applying the definition of minimax decision rule. Observe that a minimax decision rule attains an infimum risk over the space of all possible decision rules. This space can be large and, as a result, it is generally complicated to directly perform calculations for all the elements in it. Theorem 4 allows one to perform calculations for a single decision rule and prove that this “candidate” is minimax.

In the crowdsourcing problem, Theorem 1 obtains a Bayes decision rule $\delta^{*}_{c}$ for each prior $c=P(\theta=1)$. Hence, if some $\delta^{*}_{c}$ has constant risk, it follows from Theorem 4 that $\delta^{*}_{c}$ is minimax. Observe that

[TABLE]

Hence, by combining Theorems 1 and 4 one obtains

Lemma 3

If

[TABLE]

then $\delta^{*}_{c}$ has constant risk and is minimax.

In order to find a $\delta^{*}_{c}$ such as in Lemma 3, one must choose an appropriate value for $c$. It follows from Assumption 1 that ${\mathbf{Y}}|\theta=0$ is identically distributed to $1-{\mathbf{Y}}|\theta=1$. This fact is used from the second to the third line of the following derivation:

[TABLE]

Moreover,

[TABLE]

Observe that the last term in Equation 2.2 equals to the last term in Equation 2.2 when $\log(\frac{c}{1-c})=-\log(\frac{c}{1-c})=0$. This occurs when $c=\frac{1}{2}$. Theorem 3 is proved from combining Lemma 3 and Equations 2.2 and 2.2. The coin flip, $U$, was essential in the derivation of the minimax decision rule, since it allows one to divide the risk equally between $\theta=1$ and $\theta=0$ when $\sum_{i=1}^{n}w_{i}^{*}Y_{i}=C^{*}$.

2.3 Least favourable prior

The crowdsourcing problem can also be used to illustrate a strategy for finding a least favorable prior.

Definition 1

Let $P^{*}$ be a prior distribution associated with a Bayes rule, $\delta^{*}$. $P^{*}$ is said to be a least favorable prior if, for every other prior, $P$, and associated Bayes rule, $\delta$,

[TABLE]

In words, the least favorable prior leads one to use a Bayes rule with the largest Bayes risk. In this sense, the least favorable prior is such that it makes it the hardest to obtain a desirable result in the decision problem. In order to find a least favorable prior, one can use the following theorem:

Theorem 5** ((Parmigiani and Inoue, 2009))**

A prior associated with a Bayes rule with constant risk is a least favorable prior.

One can combine Theorems 3 and 5 to conclude that $P^{*}(\theta=1)=1/2$ is the least favorable prior in the crowdsourcing problem. Such a use of Theorem 5 is usually easier than directly exploring the full space of prior distributions for a least favorable prior.

3 Experts’ accuracies are unknown

Although the previous section considers the experts’ accuracies to be known, one often is uncertain about these quantities. In the following subsections, we consider the case in which the experts’ accuracies are unknown parameters and use this case to illustrate more proof strategies. Formally, in subsections 3.1 and 3.2 the parameter space is

[TABLE]

3.1 Bayes decision rule

Considering the generalization of the parameter space, not every prior allows an analytical derivation of the posterior distribution. In order to be able to derive the Bayes decision rules in this case, we consider only priors of the following form:

Assumption 2

$\theta,\gamma_{1},\ldots,\gamma_{n}$* are independent.* 2. 2.

$\gamma_{i}\sim\text{Beta}(\alpha_{i},\beta_{i})$* ($\alpha_{i}$ and $\beta_{i}$ are hyperparameters chosen a priori).*

Assumption 2.1 breaks the derivation of the posterior into smaller parts. Assumption 2.2 yields a conjugate distribution for Bernoulli trials, such as the crowdsourcing problem.

Under Assumption 2, the Bayes decision rules derived under unknown expert accuracies can be compared to the ones derived when the accuracies are known. Notation 3 is analogous to notation 2 and is useful for concisely presenting these decision rules.

Notation 3

$w_{i}^{**}:=2\log\left(\frac{\alpha_{i}}{\beta_{i}}\right)$* and $C^{**}:=\log\left(\frac{1-c}{c}\right)+\sum_{i=1}^{n}\frac{w_{i}^{**}}{2}$.*

Theorem 6

For each $c\in[0,1]$ the following decision rule is a Bayes decision rule

[TABLE]

Observe that $\delta^{**}_{c}$ is analogous to $\delta^{*}_{c}$, in Theorem 1. That is, if a weighted sum of the experts’ opinions is greater than a given cutoff, then $\delta^{**}_{c}$ decides for $1$. Furthermore, the weights in each decision rule also admit similar interpretations. It follows from Assumption 2 that $E[\gamma_{i}]=\frac{\alpha_{i}}{\alpha_{i}+\beta_{i}}$ is the expected accuracy of expert $i$. As a result, $\delta^{**}_{c}$ can be obtained from $\delta^{*}_{c}$ by substituting every instance of $\gamma_{i}$ by $E[\gamma_{i}]$. Therefore, the larger the expected accuracy of expert $i$, the more this expert is decisive in determining the value of $\delta^{**}_{c}$.

Example 1

The Bayes decision rule that is obtained in Theorem 6 is affected by the prior distribution for the experts’ accuracies, $\gamma_{1},\gamma_{2}$ and $\gamma_{3}$. For example, consider the following prior specifications:

• •

Prior 1: $\gamma_{1}\sim\mbox{Beta}(5,2);\gamma_{2},\gamma_{3}\sim\mbox{Beta}(1,1)$. The decision-maker believes that the first expert is accurate, and he is indifferent about the others.

• •

Prior 2: $\gamma_{1},\gamma_{2}\sim\mbox{Beta}(5,2);\gamma_{3}\sim\mbox{Beta}(1,1)$. The decision-maker believes that the first two experts are accurate, and he is indifferent about the third.

• •

Prior 3: $\gamma_{1},\gamma_{2}\sim\mbox{Beta}(5,2);\gamma_{3}\sim\mbox{Beta}(2,5)$. The decision-maker believes that the first two experts are accurate, and that the third expert is a spammer.

Furthermore, consider that, for every one of these priors, $c=\frac{1}{2}$.

Table 2 presents the Bayes decisions that are obtained for each of the prior that were discussed above. This table illustrates that, by choosing a prior that properly encodes one’s uncertainty, it is possible to obtain a decision rule that rationally combine the experts’ opinions about $\theta$. For example, according to Prior $1$, expert $1$ is the only accurate one. As a result, the Bayes decision rule always agrees with expert $1$, whatsoever the other opinions are. Similarly, under Prior $2$ the only accurate experts are $1$ and $2$. In this case, the Bayes decision agrees with experts $1$ and $2$ when they agree with each other. When experts $1$ and $2$ disagree, there exist Bayes decision rules that agree with either one of them. Finally, according to Prior $3$ experts $1$ and $2$ are accurate and expert $3$’s opinion is frequently incorrect. As a result, the Bayes decision rule agrees with experts $1$ and $2$ when they agree, and decides (exactly) against expert $3$ when experts $1$ and $2$ disagree.

In order to prove Theorem 6, one can solve the extensive form of the crowdsourcing problem, as presented in Theorem 2. In this case, Lemma 2 can be applied. Therefore, it is sufficient to derive the posterior distribution for $\theta$.

Let $\boldsymbol{\gamma}=(\gamma_{1},\ldots,\gamma_{n})$. The posterior distribution of $\theta$ is given by

[TABLE]

It follows from Equations 4 and 5 that

[TABLE]

Observe that $g_{i}(y_{i}):=\log\left(\frac{\Gamma(\alpha_{i}+y_{i})\Gamma(\beta_{i}+1-y_{i})}{\Gamma(\alpha_{i}+1-y_{i})\Gamma(\beta_{i}+y_{i})}\right)$ is the contribution of expert $i$ to the decision rule. This expression can be simplified: $g_{i}(0)=\log\left(\frac{\beta_{i}}{\alpha_{i}}\right)$ and $g_{i}(1)=\log\left(\frac{\alpha_{i}}{\beta_{i}}\right)$. As a result, Equation 3.1 can be simplified: applying Notation 3 to Equation 3.1, one obtains

[TABLE]

Similarly,

[TABLE]

Theorem 6 follows from applying Equations 7 and 8 to Corollary 1 and Lemma 2.

3.2 Minimax decision rule

Observe that the parameter space in this section is very conservative, in the sense of including a wide variety of possibilities for the experts’ accuracies. For example, when every $\gamma_{i}=\frac{1}{2}$, then the experts’ opinions are distributed according to a $\mbox{Bernoulli}(\frac{1}{2})$, that is, their opinions do not bring much information about $\theta$. Therefore, it might not be surprising to find a minimax decision rule that does not depend on the experts’ opinions. In this subsection we show that a coin flip, as in Definition 2, is a minimax decision rule.

Definition 2

The coin flip is a decision rule, $\delta^{cf}$, such that, for every ${\mathbf{y}}$,

[TABLE]

Theorem 7

The coin flip, $\delta^{cf}$, is a minimax decision rule.

The reason why a coin flip is, in the minimax sense, a good decision is that, for every expert, $\Theta$ includes the possibility that the expert is accurate and that he is inaccurate. As a result, if the decison-maker decides to use the experts’ opinions, then there exists a possibility that he is performing poorly. The minimax perspective is conservative with this respect and leads the decision-maker to avoid this undesirable possibility by ignoring the experts’ opinions.

In order to prove that the coin flip is a minimax decision rule, we use Theorem 4 and apply the same strategy as when the experts’ accuracies were known. Assume that, for every $i$, $\alpha_{i}=\beta_{i}=1$ and ${\mathbb{P}}(\theta=1)=1/2$. This assumption corresponds to a uniform distribution over $\Theta$. In this case, Equations 4 and 5 simplify to

[TABLE]

In other words, for every ${\mathbf{y}}$, ${\mathbb{P}}(\theta=1\!\>\lvert\>\!{\mathbf{y}})={\mathbb{P}}(\theta=0\!\>\lvert\>\!{\mathbf{y}})=\frac{1}{2}$. Therefore, when the distribution over $\Theta$ is uniform, the (marginal) distribution a posteriori for $\theta$ is the same as the distribution a priori for $\theta$. Since the decision-maker is, a priori, indifferent between $\theta=1$ and $\theta=0$ (${\mathbb{P}}(\theta=0)={\mathbb{P}}(\theta=1)$), he is also indifferent between these options a posteriori, no matter what the experts’ opinions are. It follows from Lemmas 1 and 2 that every decision rule is a Bayes decision rule. In particular, according to Corollary 1, the coin flip is a Bayes decision rule. Also observe that, for every $\boldsymbol{\gamma}\in[0,1]^{n}$,

[TABLE]

Therefore, $\delta^{cf}$ is a Bayes decision rule with constant risk. It follows from Theorem 4 that the coin flip is minimax.

The coin flip is not the unique minimax rule. If a decision rule, $\delta^{*}$, is such that, for every ${\mathbf{y}}$, $\delta^{*}({\mathbf{y}},U)$ is either $U$ or $1-U$, then the risk function of $\delta^{*}$ is the same as that of $\delta^{cf}$. As a result, such a $\delta^{*}$ is minimax. Indeed, all of the minimax decision rules are of the form described above. Let $\delta^{*}$ be a decision rule such that, for some ${\mathbf{y}}$, $\delta^{*}({\mathbf{y}},U)$ is neither $U$ or $1-U$. Since the image of $\delta^{*}$ is $\{0,1\}$, then $\delta^{*}({\mathbf{y}},U)$ is some constant $d_{{\mathbf{y}}}$. Let $\theta^{*}=1-d_{{\mathbf{y}}}$. Let $\gamma^{*}$ be such that, for every $i$, $\gamma_{i}^{*}=1$, if $y_{i}=\theta^{*}$, and $\gamma_{i}^{*}=0$, otherwise. Observe that, $P({\mathbf{Y}}={\mathbf{y}}|\gamma^{*},\theta=\theta^{*})=1$, that is, when $\theta=\theta^{*}$ and the expert precisions are $\gamma^{*}$, the decision is $d_{{\mathbf{y}}}=1-\theta^{*}$ and is incorrect with probability $1$. As a result, the supremum of the risk of this decision rule is $1$ and it is not minimax.

The above derivation shows that the coin flip can also be justified in the Bayesian framework, when one adopts a uniform prior over $\Theta$, the least favorable prior (definition 1 and theorem 5). In this case, one not only is indifferent between $\theta=0$ and $\theta=1$ a priori but also assumes that the experts’ opinions bring no information about $\theta$, that is, $P(\theta=1|{\mathbf{y}})=\frac{1}{2}$, for every ${\mathbf{y}}$ . This induces the coin flip to be as good as every other decision rule. However, for other priors over $\Theta$, the experts’ opinions generally bring more information about $\theta$. In such cases, using this information is better than the coin flip and the latter is not a Bayes decision rule. By including prior information about the accuracy of the experts, the Bayesian decision rules are generally less conservative than the minimax decision rule.

3.3 Frequentist a priori information

Despite the conclusion in Theorem 3.2 that a coin flip is a minimax decision rule, this decision rule is generally understood as a bad one. One way to solve this apparent contradiction is to add a priori information to the crowdsourcing problem by restricting the parameter space. Often, it is reasonable to assume that every expert is better than a coin flip by at least some known threshold. For instance, for the cosmology problem discussed in the introduction, the model from Izbicki and Stern (2013a) provides the estimates for each of the precisions $\gamma_{i}$ shown in Figure 1, which are in agreement with such assumption.

Formally, this assumption can be written as

Assumption 3

The parameter space of the crowdsourcing problem is

[TABLE]

for some fixed $\epsilon$ such that $0<\epsilon<\frac{1}{2}$.

Theorem 8

Under Assumption 3, $\delta_{M}$ (corollary 2) is a minimax decision rule.

That is, if every expert is better than a coin flip by at least some known threshold, then the majority rule is once again minimax.

In order to prove Theorem 8, one cannot use the strategy from Section 2.2. This happens because, although $\delta_{M}$ is a Bayes decision rule, it doesn’t have constant risk. Indeed, the higher the accuracy of the experts, the smaller the risk of $\delta_{M}$. In order to prove that $\delta_{M}$ is minimax, we use the following standard strategy for finding minimax rules:

Find a decision rule, $\delta_{M}$, that one believes to be minimax. 2. 2.

Derive an upper bound on the minimax risk using $\delta_{M}$, that is:

[TABLE] 3. 3.

Derive a lower bound on the minimax risk by considering a subset of the parameter space. That is, if $\Theta^{*}_{0}\subset\Theta^{*}$, then

[TABLE]

If the upper bound matches the lower bound, then $\delta_{M}$ is minimax:

Theorem 9** ((Tsybakov, 2008))**

If there exist $\delta_{M}$ and $\Theta^{*}_{0}\subset\Theta^{*}$ such that

[TABLE]

then $\delta_{M}$ is a minimax decision rule.

Since all experts are better than a coin flip on $\Theta^{*}$, a reasonable guess for a decision rule is the majority rule, $\delta_{M}$, that is defined in Corollary 2. Indeed, recall from Theorem 3 and Corollary 2 that, if the expert accuracies are known and are all equally greater than $\frac{1}{2}$, then the majority rule is a minimax decision rule.

Following the second step above, we bound the value of $\sup_{\xi\in\Theta^{*}}\mbox{ }R(\delta_{M},\xi)$. Observe that

[TABLE]

The last inequality follows from observing that {\mathbb{P}}\left(\bar{{\mathbf{Y}}}>\frac{1}{2}\!\>\Big{\lvert}\>\!\theta=0,\boldsymbol{\gamma}\right) is strictly decreasing over each $\gamma_{i}$ (the higher the accuracies of the experts, the lower the value of $\bar{{\mathbf{Y}}}$ when $\theta=0$). Therefore, this probability is maximized by picking, for each $i$, the lowest value of $\gamma_{i}$ that is allowed on $\Theta^{*}$. Using the above derivation, conclude that

[TABLE]

The third step in the derivation strategy obtains a lower bound on the minimax risk by considering some subspace $\Theta_{0}^{*}$ of $\Theta^{*}$. This subset is often called a reduction scheme (Tsybakov, 2008). Consider

[TABLE]

It follows from Theorem 3 and Corollary 2 that $\delta_{M}$ is a minimax decision rule over $\Theta^{*}_{0}$ and has constant risk. Therefore,

[TABLE]

Finally, one obtains that $\sup_{\xi\in\Theta^{*}}R(\delta_{M},\xi)\leq\inf_{\delta}\sup_{\xi\in\Theta_{0}^{*}}\mbox{ }R(\delta,\xi)$ by combining Equations 9 and 10. Therefore, one concludes from Theorem 9 that the majority rule, $\delta_{M}$, is a minimax decision rule considering $\Theta^{*}$ in place of the original $\Theta$. Notice that a similar argument holds in the case where the accuracy threshold, $\epsilon$, varies among experts. Let $\epsilon_{i}$ denote the accuracy threshold for expert $i$. In this case, the decision rule $\delta^{*}_{\frac{1}{2}}$ from Theorem 3 that uses $\gamma_{i}=\frac{1}{2}+\epsilon_{i}$ is a minimax decision rule. The minimax decision rule incorporates the information about the experts’ precisions that is expressed in the choice of the parameter space.

4 Conclusions

We illustrate a set of standard techniques for finding Bayes and minimax decision rules in an elementary crowdsourcing example. The following techniques are illustrated:

• •

Theorem 2: Rather than directly minimizing ${\mathbb{E}}_{\theta}[R(\delta({\mathbf{Y}}),\theta)]$, the Bayes rule can be found by minimizing ${\mathbb{E}}[L(a,\theta)\!\>\lvert\>\!{\mathbf{y}}]$ for each ${\mathbf{y}}$.

• •

Theorem 4: Finding a Bayes rule with constant risk is a way to find a minimax rule, and the prior associated to this rule is the least favorable prior.

• •

Theorem 9: A minimax rule can be found by providing an upper bound and a lower bound of the minimax risk and showing that they match. An upper bound on the minimax risk can be found by using the supremum of the risk of a reasonable (candidate) decision rule. A lower bound on the minimax risk can be found using a reduction scheme.

The crowdsourcing problem was also used to illustrate the differences between the Bayes decision rules and the minimax decision rules. Since a minimax decision rule considers the worst case scenario, it is more conservative than a Bayes decision rule. However, this conservatism can lead to odd minimax decision rules, such as the coin flip in Theorem 7. This situation is overcome by adding more a priori information to the model. Such information can be incorporated either as a prior distribution over expert accuracies, in the Bayesian framework, or as restrictions on the parameter space, in the frequentist framework.

The second author presented this material in a master’s level decision theory class. The content was delivered in $2$ hours and the students were interested by the example, specially because of its practical appeal in comparison to the standard normal and binomial examples. They were particularly surprised by some situations then introduced. For example, they were curious about the case in which minimax procedures reduce to flipping a coin and about the situation in which every decision rule was a Bayes decision rule.