Does Confidence Reporting from the Crowd Benefit Crowdsourcing Performance?

TL;DR

This paper investigates whether confidence reporting improves crowdsourcing classification accuracy and finds that allowing workers to reject microtasks without reporting confidence yields better performance.

Contribution

The study introduces an aggregation method with optimized weighting and demonstrates that confidence reporting does not enhance classification performance in crowdsourcing.

Findings

Confidence reporting does not improve classification accuracy.

Reject option without confidence reporting is optimal.

Weighted majority voting maximizes crowd performance.

Abstract

We explore the design of an effective crowdsourcing system for an $M$-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final classification decision. We consider the scenario where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond to binary microtasks. Additionally, the workers report quantized confidence levels when they are able to submit definitive answers. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classification performance. We obtain a couterintuitive result that the classification performance does not benefit from workers reporting quantized confidence. Therefore, the crowdsourcing system designer should employ the reject option without requiring confidence reporting.

Figures

Equations

$$e_D=\arg \underset{e_j\in D}{\max }\left\{\sum _{w=1}^W{W}_w{I}_{D_w}⟨e_j⟩\right\},j=1,2,\dots ,M,$$

$$\begin{array}[]{l}\text{maximize}\ \ {E_{C}}\left[{{\mathbb{W}}}\right]\\ \text{subject to}\ \ {E_{O}}\left[{{\mathbb{W}}}\right]={K}\end{array}$$

$$W_w={\mu }^{-n},$$

$$\hat{\mu }=\frac{1}{W-ϵ}\sum _{w=1}^W\sum _{i=1}^N\frac{l_{w,i}-\frac{1}{2}}{LI(w)},$$

$$\displaystyle P_{c}=\Big{[}\frac{1}{2}$$

$$\displaystyle+\frac{1}{4}\sum\limits_{S^{\prime}}{\binom{W}{\mathbb{Q}}}\left({F\left(\mathbb{Q}\right)-F^{\prime}\left(\mathbb{Q}\right)}\right)\Big{]}^{N},$$

$$F(\mathbb{Q})=m^{q_0}\prod _{n=1}^N{\left(1-\mu \right)}^{q_{-n}}{\mu }^{q_n}{\left(C_{N-1}^{n-1}{\left(1-m\right)}^n{m}^{N-n}\right)}^{q_{-n}+q_n}$$

$$F^{\prime }(\mathbb{Q})=m^{q_0}\prod _{n=1}^N{\left(1-\mu \right)}^{q_n}{\mu }^{q_{-n}}{\left(C_{N-1}^{n-1}{\left(1-m\right)}^n{m}^{N-n}\right)}^{q_{-n}+q_n}.$$

$$E_C[\mathbb{W}]$$

$$=E_{p,\rho ,l}\left[\sum _{w=1}^W\sum _{n=0}^N\sum _{N_n\in [N]}\prod _{i\in N_n}{W}_w(1-p_{w,i}){\rho }_{w,i}\prod _{j\in [N]-N_n}{p}_{w,j}|l_{w,i}\right]$$

$$A_w(p_{w,i})=\sum _{n=0}^N{E}_{\rho ,l}\left[W_w\prod _{i\in N_n}{\rho }_{w,i}|l_{w,i}\right]P_{\lambda }(n),$$

$$A_w(p_{w,i})=\sum _{n=0}^N{E}_{\rho ,l}\left[W_w\prod _{i\in N_n}{\rho }_{w,i}|l_{w,i}\right]\sqrt{P_{\lambda }(n)}\sqrt{P_{\lambda }(n)}$$

$$\le \sqrt{\sum _{n=0}^N{E}_{\rho ,l}^2\left[W_w\prod _{i\in N_n}{\rho }_{w,i}|l_{w,i}\right]P_{\lambda }(n)}\sqrt{\sum _{n=0}^N{P}_{\lambda }(n)}.$$

$$E_{\rho ,l}\left[W_w\prod _{i\in N_n}{\rho }_{w,i}|l_{w,i}\right]\sqrt{P_{\lambda }(n)}={\alpha }_w(p_{w,i})\sqrt{P_{\lambda }(n)},$$

$$E_{\rho ,l}\left[W_w\prod _{i\in N_n}{\rho }_{w,i}|l_{w,i}\right]={\alpha }_w(p_{w,i}).$$

$$E_p[A_w(p_{w,i})]\le \underset{p_{w,i}}{\int }{\alpha }_w(p_{w,i})\mathrm{Pr}\left(p_{w,i}=x\right)dx$$

$$\le \sqrt{\underset{p_{w,i}}{\int }{\alpha }_w^2(p_{w,i})\mathrm{Pr}\left(p_{w,i}=x\right)dx}\sqrt{\underset{p_{w,i}}{\int }\mathrm{Pr}\left(p_{w,i}=x\right)dx}.$$

$${\alpha }_w(p_{w,i})\sqrt{\mathrm{Pr}\left(p_{w,i}=x\right)}=\beta \sqrt{\mathrm{Pr}\left(p_{w,i}=x\right)},$$

$$W_w=\frac{\beta }{E_{\rho ,l}[\prod _{i\in N_n}{\rho }_{w,i}|l_{w,i}]}=\frac{\beta }{{\mu }^n}.$$

$$E_O\left[\mathbb{W}\right]$$

$$=\sum _{w=1}^W\sum _{n=0}^N\beta {\mu }^{-n}{2}^{N-n}\left(\genfrac{}{}{0ex}{}{N}{n}\right){\left(1-m\right)}^n{m}^{N-n}$$

$$=W\beta {\left(\frac{1-m}{\mu }+2m\right)}^N=K.$$

Full text

Classification, crowdsourcing, distributed inference, information fusion, reject option, confidence reporting

††copyright: acmcopyright††conference: SocialSens’2017; Apr. 21 2017; Pittsburgh, PA, USA††isbn: 978-1-4503-4977-2/17/04††price: 15.00††doi: http://dx.doi.org/10.1145/3055601.3055607††ccs: Human-centered computing Social network analysis

1. Introduction

Crowdsourcing provides a new framework to utilize distributed human wisdom to solve problems that machines cannot perform well, like handwriting recognition, paraphrase acquisition, audio transcription, and photo tagging (Paritosh et al., 2011; Burrows et al., 2013; Fan et al., 2015). Despite the successful applications of crowdsourcing, the relatively low quality of output is a key challenge (Ipeirotis et al., 2010; Allahbakhsh et al., 2013; Mo et al., 2013).

Several methods have been proposed to deal with the aforementioned problems (Karger et al., 2011; Vempaty et al., 2014; Yue et al., 2014; Sanchez-Charles et al., 2014; Varshney et al., 2014; Quinn and Bederson, 2011; Zhang and van der Schaar, 2012; Hirth et al., 2013). A crowdsourcing task is decomposed into microtasks that are easy for an individual to accomplish, and these microtasks could be as simple as binary distinctions (Karger et al., 2011). A classification problem with crowdsourcing, where taxonomy and dichotomous keys are used to design binary questions, is considered in (Vempaty et al., 2014). In our research group, we employed binary questions and studied the use of error-control codes and decoding algorithms to design crowdsourcing systems for reliable classification (Varshney et al., 2014; Vempaty et al., 2014). A group control mechanism where the reputation of the workers is taken into consideration to partition the crowd into groups is presented in(Quinn and Bederson, 2011; Zhang and van der Schaar, 2012). Group control and majority voting are compared in (Hirth et al., 2013), which reports that majority voting is more cost-effective on less complex tasks.

In past work on classification via crowdsourcing, crowd workers were required to provide a definitive yes/no response to binary microtasks. Crowd workers may be unable to answer questions for a variety of reasons such as lack of expertise. As an example, in mismatched speech transcription, i.e., transcription by workers who do not know the language, workers may not be able to perceive the phonological dimensions they are tasked to differentiate (Jyothi and Hasegawa-Johnson, 2015). In recent work, we have investigated the design of the optimal aggregation rule when the workers have a reject option so that they are unable to or choose not to respond (Li et al., 2017).

The possibility of using confidence scores to improve the quality of crowdsourced labels was investigated in (Kazai, 2011). An aggregation method using confidence scores to integrate labels provided by crowdsourcing workers was developed in (Oyama et al., 2013). A payment mechanism was proposed for crowdsourcing systems with a reject option and confidence score reporting (Shah and Zhou, 2014). Indeed, confidence reporting can be useful for estimating the quality of the provided responses and possibly yield better outcomes when the aggregation is not optimal. However, potential crowdsourcing performance improvement with an optimal aggregation rule resulting from confidence reporting has not yet been investigated. As is studied in this paper, when an optimal aggregation rule is developed, confidence reporting does not help to improve the performance.

In this paper, we further consider the problem investigated in (Li et al., 2017) by studying the scenario when the workers include their confidence levels in their responses. The main contribution of this paper is the counterintuitive finding that the confidence scores of the crowd do not play a role in the optimal aggregation rule. The weight assignment scheme to ensure the maximum weight for the correct class is the same as that when there is no confidence reporting. Although confidence reporting can provide useful information for estimating the quality of the crowd, the noise introduced due to categorization of confidence makes the estimation less accurate. Since the estimation result is essential for aggregation, confidence reporting may cause performance degradation.

2. Crowdsourcing Task with a Reject Option

Consider the situation where $W$ workers take part in an $M$-ary object classification task. Each worker is asked $N$ simple binary questions, termed as microtasks, and the worker’s answer to a single microtask is conventionally represented by either “1” (Yes) or “0” (No), which eventually lead to a classification decision among the $M$ classes. We assume independent microtask design and, therefore, we have $N=\left\lceil{{{\log}_{2}}M}\right\rceil$ independent microtasks of equal difficulty. The workers submit responses that are combined to give the final decision. Here, we consider the microtasks to be simple binary questions and the worker’s answer to a single microtask is conventionally represented by either “1” (Yes) or “0” (No) (Vempaty et al., 2014; Rocker et al., 2007). Thus, the $w$th worker’s ordered answers to all the microtasks form an $N$-bit word, which is denoted by ${\bf a}_{w}$. Let ${\bf a}_{w}(i)$, $i\in\{1,2,\dots,N\}$ represent the $i$th bit in this vector.

In our previous work (Li et al., 2017), we considered a general problem setting where the worker has a reject option of skipping the microtasks. We denote this skipped answer as $\lambda$, whereas the “1/0” (Yes/No) answers are termed as definitive answers. Due to the variability of different worker backgrounds, the probability of submitting definitive answers is different for different workers. Let $p_{w,i}$ represent the probability of the $w$th worker submitting $\lambda$ for the $i$th microtask. Similarly, let $\rho_{w,i}$ be the probability that ${\bf a}_{w}(i)$, the $i$th answer of the $w$th worker, is correct given that a definitive answer has been submitted. Due to the variabilities and anonymity of workers, we study crowdsourcing performance when $p_{w,i}$ and $\rho_{w,i}$ are realizations of certain probability distributions, which are denoted by distributions ${F_{P}}\left(p\right)$ and ${F_{\rho}}\left(\rho\right)$ respectively. The corresponding means are expressed as $m$ and $\mu$.

Let $H_{0}$ and $H_{1}$ denote the hypotheses where “0” or “1” is the true answer for a single microtask, respectively. For simplicity of performance analysis, $H_{0}$ and $H_{1}$ are assumed equiprobable for every microtask. The crowdsourcing task manager or a fusion center (FC) collects the $N$-bit words from $W$ workers and performs fusion based on an aggregation rule. We focus on finding the optimal aggregation rule and let us briefly review the results regarding the aggregation of responses from the workers for classification in our previous work (Li et al., 2017).

• Let $D=\{e_{j},j=1,2,\dots,M\}$ be the set of all the object classes, where $e_{j}$ represents the $j$th class. Based on $w$th worker’s response to the microtasks, a subset $D_{w}$ is chosen, within which the classes are associated with weight $W_{w}$ for aggregation.111If all the responses from the $w$th worker are definitive, $D_{w}$ is a singleton. Otherwise, $D_{w}$ contains multiple classes. The fusion center FC adds up the weights for every class and chooses the one with highest overall weight as the final decision $e_{D}$, which can be expressed as

[TABLE]

where $I_{{D_{w}}}\left\langle{{e_{j}}}\right\rangle$ is an indicator function which equals 1 if $e_{j}\in D_{w}$ and 0 otherwise. To derive the optimal weight $W_{w}$ for each worker, one may look into the minimization of the misclassification probability, for which a closed-form expression cannot be derived without an explicit expression for $W_{w}$. Hence, it is difficult to determine the optimal weight.

•  The $M$-ary classification task can also be split into $N$ binary hypothesis testing problems, by associating a classification decision with an $N$-bit word. Each worker votes “1” or “0” with the weight $W_{w}$ for every bit. In this case, the Chair-Varshney rule gives the optimal weight as $W_{w}={\log\frac{{{\rho_{w,i}}}}{{1-{\rho_{w,i}}}}}$ (Chair and Varshney, 1986). However, this requires the prior knowledge on $\rho_{w,i}$ for every worker, which is not available in practice.

•  We proposed a novel weighted majority voting method, which was derived by solving the following optimization problem

[TABLE]

where ${E_{C}}\left[{{\mathbb{W}}}\right]$ denotes the crowd’s average weight contribution to the correct class and ${E_{O}}\left[{{\mathbb{W}}}\right]$ denotes the average weight contribution to all the possible classes that is constrained to remain a constant $K$. Statistically, this method ensures maximum weight to the correct class and consequently maximum probability of correct classification. We showed that this method significantly outperforms the simple majority voting procedure.

In this paper, we investigate the impact of confidence reporting from the crowd on system performance. The weight assignment scheme is developed by solving problem (2) as well.

3. Crowdcouring with Confidence Reporting

We consider the case where the crowd is composed of honest workers, which means that the workers honestly observe, think, and answer the questions, give confidence levels, and skip questions that they are not confident about. We derive the optimal weight assignment for the workers and the performance of the system in a closed form. Based on these findings, we determine the potential benefits of confidence reporting in a crowdsourcing system with a reject option.

3.1. Confidence Level Reporting

In a crowdsourcing system where workers submit answers and report confidence, we define the $w$th worker’s confidence about the answer to the $i$th microtask as the probability of this answer being correct given that this worker gives a definitive answer, which is equal to $\rho_{w,i}$ as defined earlier. When $\rho_{w,i}$ is bounded as $\frac{l_{w,i}-1}{L}\leq\rho_{w,i}\leq\frac{l_{w,i}}{L}$, $l_{w,i}\in\{1,\ldots,L\}$, the $w$th worker reports his/her confidence level as $l_{w,i}$. Let $l_{w,i}$ be drawn from the distribution $l_{w,i}\sim F_{L}(l)$. Note that every worker independently gives confidence levels for different microtasks, and $L=1$ simply means that workers submit answers and do not report their confidence levels.

Assuming that a worker can accurately perceive the probability $\rho_{w,i}$ and honestly report the confidence level, intuitively it is expected that it will benefit the crowdsourcing fusion center as much more information about the quality of the crowd can be extracted. However, as the confidence is quantized, which helps the workers in determining the confidence levels to be reported, quantization noise is introduced in extracting the crowd quality from confidence reporting.

As an illustrative example, consider the problem of mismatched crowdsourcing for speech transcription, which has garnered interest in the signal processing community (Hasegawa-Johnson et al., 2015; Jyothi and Hasegawa-Johnson, 2015; Varshney et al., 2016; Liu et al., 2016; Chen et al., 2016; Kong et al., 2016). Suppose the four possibilities for a velar stop consonant to transcribe are $R=\{$ k, K, g, G$\}$. The simple binary question of “whether it is aspirated or unaspirated” differentiates between $\{$ K, G$\}$ and $\{$ k, g$\}$, whereas the binary question of “whether it is voice or unvoiced” differentiates between $\{$ g, G$\}$ and $\{$ k, K $\}$. The highest confidence level is set as $L=4$. Now suppose the first worker is a native Italian speaker. Since Italian does not use aspiration, this worker will be unable to differentiate between $\{$ k$\}$ and $\{$ K$\}$, or between $\{$ g$\}$ and $\{$ G$\}$. It would be of benefit if this worker would specify the inability to perform the task through a special symbol $\lambda$, rather than guessing randomly, and this worker answers “Yes” with confidence level 1 to the second question. Suppose the second worker is a native Bengali speaker. Since this language makes a four-way distinction among velar stops, such a worker will probably answer both questions without a $\lambda$.

In the rest of this section, we address the problem “Does the confidence reporting help crowdsourcing system performance?” by performing analyses when workers report their confidences with their definitive answers.

3.2. Optimal Weight Assignment Scheme

We determine the optimal weight $W_{w}$ for the $w$th worker in this section. We rewrite hereby the weight assignment problem

[TABLE]

where ${E_{C}}\left[{{\mathbb{W}}}\right]$ denotes the crowd’s average weight contribution to the correct class and ${E_{O}}\left[{{\mathbb{W}}}\right]$ denotes the average weight contribution to all the possible classes and remains a constant $K$. Statistically, we are looking for the weight assignment scheme such that the weight contribution to the correct class is maximized while the weight contribution to all the classes remains fixed, so as to maximize the probability of correct classification.

Proposition 3.1.

To maximize the average weight assigned to the correct classification element, the weight for $w$th worker’s answer is given by

[TABLE]

where $n$ is the number of definitive answers that the $w$th worker submits.

Proof.

See Appendix. ∎

Remark 1.

Here the weight depends on the number of questions answered by a worker. In fact, if more questions are answered, the weight assigned to the corresponding worker’s answer is larger. This is intuitively pleasing as a high-quality worker is able to answer more questions and is assigned a higher weight. Increased weight can put more emphasis on the contribution of high-quality workers in that sense and improve overall classification performance.

Remark 2.

When $L=\infty$, $\rho_{w,i}$ associated with every worker for every microtask is reported exactly. Then the Chair-Varshney rule gives the optimal weight assignment to minimize error probability (Chair and Varshney, 1986). However, human decision makers are limited in their information processing capacity and can only carry around seven categories (Miller, 1956). Thus, the largest value of $L$ is around 7 in practice.

Remark 3.

Note that the optimal weight assignment scheme is the same as in the case where the workers do not report confidence levels, i.e., $L=1$. Actually, the value of $L$ does not play any role in the weight assignment, as long as $\rho_{w,i}$ is not known exactly. Therefore, the weight assignment is universally optimal regardless of confidence reporting.

3.3. Parameter Estimation

Before the proposed aggregation rule can be used, $\mu$ has to be estimated to assign the weight for every worker’s answers. Here, we employ three approaches to estimate $\mu$. We refer to our previous work (Li et al., 2017) for training-based and majority-voting based methods to estimate $\mu$, and give an additional method using the information extracted from the workers’ reported confidence levels.

Confidence-based

Note that the reported confidence levels correspond to $\rho_{w,i}$. We collect all the values of the submitted confidence levels and obtain the estimate of $\mu$ from them. First, the $w$th worker’s confidence level for the $i$th microtask is represented by $l_{w,i}$. Considering the fact that $\frac{l_{w,i}-1}{L}\leq\rho_{w,i}\leq\frac{l_{w,i}}{L}$ if the worker submits a definitive answer, we use $\frac{l_{w,i}-\frac{1}{2}}{L}$ to approximate $\rho_{w,i}$. Let $l_{w,i}=\frac{1}{2}$ if the $w$th worker skips the $i$th microtask. We obtain the estimate of $\mu$ by

[TABLE]

where $I(w)$ denotes the number of definitive answers that $w$th worker submits.

3.4. Performance Analysis

In this section, we characterize the performance of the proposed crowdsourcing classification framework in terms of the probability of correct classification $P_{c}$. Note that we have overall correct classification only when all the bits are classified correctly.

Proposition 3.2.

The probability of correct classification $P_{c}$ in the crowdsourcing system is

[TABLE]

where $\mathbb{Q}=\left\{({{q_{-{N}}},{q_{-{{N+1}}}},\ldots{q_{{N}}}}):\sum\limits_{n=-N}^{N}{{q_{{n}}}=W}\right\}$ with natural numbers $q_{n}$ and $q_{0}$, and ${S}=\left\{{\mathbb{Q}:}{\sum\limits_{n=1}^{N}{{{\mu}^{-n}}\left({{q_{{n}}}-{q_{-{n}}}}\right)}>0}\right\},$ $S^{\prime}=\left\{{\mathbb{Q}:\sum\limits_{n=1}^{N}{{\mu^{-n}}\left({{q_{n}}-{q_{-n}}}\right)}=0}\right\},$ $\binom{W}{\mathbb{Q}}=\frac{{W!}}{{\prod\limits_{n=-N}^{N}{{q_{n}}!}}}$, and

[TABLE]

[TABLE]

Proof.

The proof is similar to the proof in our previous work (Li et al., 2017) and is, therefore, omitted for brevity. ∎

4. Simulation Results

In this section, we give the simulation results for the proposed crowdsourcing system. The workers take part in a classification task of $N=3$ microtasks. $F_{P}(p)$ is a uniform distribution denoted as $U(0,1)$.

First, we show the efficiency of the derived optimal weight assignment over the widely used simple majority voting method for crowdsourcing systems. The performance comparison is presented with the number of workers varying from 3 to 29. Here, we consider different qualities of the individual workers in the crowd which is represented by variable $\rho_{w,i}$ with a uniform distribution $U(0.6,1)$. Thus, the mean $\mu$ is 0.8, and we give simulation results when confidence reporting is not included and the estimation of $\mu$ is perfect in Fig. 1. It is observed that a larger crowd completes the classification task with higher quality. A significant performance improvement by the proposed method with a reject option compared with the simple majority voting is shown in the figure.

Since an accurate estimation of $\mu$ is essential for applying the optimal weight assignment scheme, we next focus on the estimation results of $\mu$ for the three estimation methods as discussed in the previous section. Let $F_{\rho}(\rho)$ be a uniform distribution expressed as $U(x,1)$ with $0\leq x\leq 1$, and thus we can have $\mu$ varying from 0.5 to 1. We consider that $W=20$ workers participate in the classification task with a reject option and confidence reporting.

In Fig. 2, it is observed that the training-based method has the best overall performance, which takes advantage of the gold standard questions. We can also see that the majority voting method has better performance as $\mu$ increases. This is because a larger $\mu$ means a better-quality crowd, which will lead to a more accurate result from majority voting, and consequently better estimation performance of $\mu$. When confidence is considered with $L=4$, we find that the overall estimation performance is not better than the other two methods because of quantization noise associated with confidence reporting in the estimation of $\mu$. It is also shown that the curve saturates and yields a fixed value of $\hat{\mu}=0.875$ when $\mu\geq 0.9$. This is because almost all the confidence levels submitted then are $l_{w,i}=4$ and the corresponding estimate result is exactly 0.875.

The estimation performance of the confidence-based method with multiple confidence levels is presented in Fig. 2. As is expected, a larger $L$ can help improve the estimation performance. However, it is seen that even though $L=8$, the corresponding performance is still not as good as that of the other two methods. Although we can expect estimation performance improvement as the maximum number of confidence levels $L$ increases, $L=8$ is pretty much the limit in practice due to the human inability to categorize beyond 7 levels. When the confidence-based estimation method is employed, the estimate value saturates at a certain fixed value when $\mu$ is large. Therefore, it can be concluded that the confidence-based estimation method does not provide good results.

Even though the three methods differ in performance in the estimation of $\mu$, we show in Fig. 3 the robustness of the proposed system. We observe from Fig. 2 that the majority voting based method suffers from performance degradation in the low-$\mu$ regime, while the confidence based one suffers in the high-$\mu$ regime. However, when the value of $\mu$ is low, the workers are making random guesses even when they believe that they are able to respond with definitive answers. When the value of $\mu$ is large, almost all the definitive answers submitted are correct. Therefore, in those two situations, the performance degradation in the estimation of $\mu$ is negligible. From Fig. 3, we see that system performance of the proposed system with estimation results from Fig. 2 is almost the same as with the other three estimation methods, which significantly outperforms the system where simple majority voting is employed without a reject option. However, if a significant performance degradation in the estimation of $\mu$ occurs outside the two aforementioned regimes, overall classification performance loss is expected. For example, consider the case where $\mu$ is 0.8 while $\hat{\mu}$ is 0.5, and $N=5$, then $P_{c}=0.8$. However, the actual $P_{c}$ equals 0.89 when $\mu$ is estimated with an acceptable error.

5. Conclusion

We have studied a novel framework of crowdsourcing system for classification, where an individual worker has the reject option and can skip a microtask if he/she has no definitive answer, and gives definitive answers with quantized confidence. We presented an aggregation approach using a weighted majority voting rule, where each worker’s response is assigned an optimized weight to maximize the crowd’s classification performance. However, we showed that reporting of confidence by the crowd does not benefit classification performance. One is advised to adopt the reject option without confidence indication from the workers as it does not improve classification performance and may degrade performance in some cases.

APPENDIX

To solve problem (3), we need $E_{C}[\mathbb{W}]$ and $E_{O}[\mathbb{W}]$. First, the $w$th worker can have weight contribution to $E_{C}[\mathbb{W}]$ only if all his/her definitive answers are correct. Thus, we have the average weight assigned to the correct element as

[TABLE]

where $[N]$ denotes $\{1,\ldots,N\}$ and $N_{n}\in[N]$ with cardinality $n$. Given a known $w$th worker, i.e., $p_{w,i}$ is known, we write

[TABLE]

where $P_{\lambda}(n)=\sum\limits_{N_{n}\in[N]}\prod\limits_{i\in N_{n}}{(1-p_{w,i})\prod\limits_{j\in[N]-N_{n}}p_{w,j}}.$

Note that ${\sum\limits_{n=0}^{N}{P_{\lambda}(n)}}=1$, and then (8) is upper-bounded using Cauchy-Schwarz inequality as follows:

[TABLE]

Also note that equality holds in (9) only if

[TABLE]

where $\alpha_{w}$ is a positive quantity independent of $n$, which might be a function of $p_{w,i}$, and

[TABLE]

Note that $\displaystyle{\int\limits_{p_{w,i}}\!\!{F_{p}({p_{w,i}=x})dx}}=1$, and similarly we write

[TABLE]

The equality (12) holds only if

[TABLE]

WHERE $\beta$ is a positive constant independent of $p_{w,i}$, and we conclude that $\alpha_{w}$ is also a positive quantity independent of $p_{w,i}$. Then from (11), we have ${E_{\rho,l}\left[{{W_{w}}{\prod\limits_{i\in N_{n}}\rho_{w,i}}}|l_{w,i}\right]}=\beta$. Since ${\prod\limits_{i\in N_{n}}\rho_{w,i}}$ is the product of $n$ variables, its distribution is not known a priori. A possible solution to weight assignment is a deterministic value given by $W_{w}E_{\rho,l}[{\prod\limits_{i\in N_{n}}\rho_{w,i}}|l_{w,i}]=\beta$ and, therefore, we can write the weight as

[TABLE]

Then, we can express the crowd’s average weight contribution to all the classes defined in (3) as

[TABLE]

Thus, $\beta$ and the weight can be obtained accordingly. Note that the weight derived above has a term that is common for every worker. Since the voting scheme is based on comparison, we can ignore this factor and have the normalized weight as $W_{w}={\mu}^{-n}$.

Acknowledgements.

This work was supported in part by the Army Research Office under Grant W911NF-14-1-0339 and in part by the National Science Foundation under Grant ENG-1609916.