Graph Mining Meets Crowdsourcing: Extracting Experts for Answer Aggregation

TL;DR

This paper introduces a graph-mining approach to identify reliable experts within crowdsourcing tasks, enhancing answer aggregation accuracy especially when non-experts dominate, and demonstrates improved performance over existing methods.

Contribution

It proposes the novel concept of 'expert core' and efficient algorithms for extracting it, improving answer aggregation in crowdsourcing by leveraging expert identification.

Findings

Expert core extraction improves answer accuracy.

Algorithms outperform state-of-the-art methods.

Theoretical justification supports the approach.

Abstract

Aggregating responses from crowd workers is a fundamental task in the process of crowdsourcing. In cases where a few experts are overwhelmed by a large number of non-experts, most answer aggregation algorithms such as the majority voting fail to identify the correct answers. Therefore, it is crucial to extract reliable experts from the crowd workers. In this study, we introduce the notion of "expert core", which is a set of workers that is very unlikely to contain a non-expert. We design a graph-mining-based efficient algorithm that exactly computes the expert core. To answer the aggregation task, we propose two types of algorithms. The first one incorporates the expert core into existing answer aggregation algorithms such as the majority voting, whereas the second one utilizes information provided by the expert core extraction algorithm pertaining to the reliability of workers. We then give a theoretical justification for the first type of algorithm. Computational experiments using synthetic and real-world datasets demonstrate that our proposed answer aggregation algorithms outperform state-of-the-art algorithms.

Tables

Table 1. Table 1: A small example of crowdsourced single-answer multiple-choice questions.

	$𝒲\backslash 𝒬$	$q_1$	$q_2$	$q_3$	$q_4$	$q_5$	$q_6$	$q_7$	$q_8$
	correct answer	D	C	D	E	B	C	A	E
\ldelim{327pt[Experts]	$w_1$	D	C	D	E	B	C	A	E
	$w_2$	D	C	B	E	B	C	A	C
	$w_3$	D	C	D	D	B	C	A	E
\ldelim{345pt[Non-experts]	$w_4$	C	B	A	A	E	C	D	B
	$w_5$	A	B	E	A	B	E	E	C
	$w_6$	C	A	B	E	B	B	A	C

Table 2. Table 2: Real datasets used in our experiments. The last column gives the best accuracy rate (i.e., the fraction of the number of correct answers) among the workers.

Dataset	$m$	$n$	$s$	Best
Chinese	24	50	5	0.79
English	30	63	5	0.70
IT	25	36	4	0.84
Medicine	36	45	4	0.92
Pokémon	20	55	6	1.00
Science	20	111	5	0.85

Table 3. Table 3: Results of answer aggregation algorithms for synthetic datasets, where n = 100 𝑛 100 n=100 and m = 50 𝑚 50 m=50 . Each cell gives the average and standard deviation of the accuracy rate over 100 data realizations. For each setting, the best average accuracy rate is written in bold.

$(p_{\mathrm{ex}},n_{\mathrm{ex}})$	Top-2	Ex-MV	Ex-GLAD	Ex-Hyper-MV	MV	GLAD	Hyper-MV
$(0.7,2)$	0.519 $\pm$0.218	0.285 $\pm$0.079	0.277 $\pm$0.098	0.276 $\pm$0.079	0.273 $\pm$0.058	0.264 $\pm$0.092	0.272 $\pm$0.062
$(0.7,4)$	0.892 $\pm$0.047	0.388 $\pm$0.120	0.563 $\pm$0.212	0.513 $\pm$0.120	0.348 $\pm$0.070	0.549 $\pm$0.212	0.514 $\pm$0.131
$(0.7,6)$	0.911 $\pm$0.040	0.624 $\pm$0.247	0.886 $\pm$0.073	0.782 $\pm$0.247	0.426 $\pm$0.066	0.865 $\pm$0.083	0.755 $\pm$0.106
$(0.8,2)$	0.750 $\pm$0.097	0.320 $\pm$0.110	0.300 $\pm$0.131	0.405 $\pm$0.110	0.284 $\pm$0.069	0.303 $\pm$0.139	0.380 $\pm$0.126
$(0.8,4)$	0.956 $\pm$0.026	0.783 $\pm$0.259	0.896 $\pm$0.138	0.866 $\pm$0.259	0.373 $\pm$0.069	0.814 $\pm$0.160	0.813 $\pm$0.095
$(0.8,6)$	0.958 $\pm$0.028	0.984 $\pm$0.018	0.982 $\pm$0.020	0.969 $\pm$0.018	0.484 $\pm$0.065	0.950 $\pm$0.031	0.962 $\pm$0.030
$(0.9,2)$	0.879 $\pm$0.048	0.356 $\pm$0.172	0.384 $\pm$0.214	0.702 $\pm$0.172	0.290 $\pm$0.067	0.353 $\pm$0.180	0.704 $\pm$0.146
$(0.9,4)$	0.991 $\pm$0.014	0.989 $\pm$0.015	0.991 $\pm$0.014	0.988 $\pm$0.015	0.416 $\pm$0.063	0.941 $\pm$0.091	0.982 $\pm$0.021
$(0.9,6)$	0.989 $\pm$0.014	0.998 $\pm$0.005	0.998 $\pm$0.006	0.996 $\pm$0.005	0.550 $\pm$0.073	0.984 $\pm$0.016	0.997 $\pm$0.007

Equations

$$\mathrm{Pr}[l_{wq}=c]=\{\begin{array}{ll}{p}_{\mathrm{ex}}& (c=a_q,w\in \mathcal{E}),\\ \frac{1-p_{\mathrm{ex}}}{s-1}& (c\ne a_q,w\in \mathcal{E}),\\ 1/s& (w\in \mathcal{W}\setminus \mathcal{E}).\end{array}$$

$$\prod _{v\in W\setminus \{u\}}\sum _{i=\tau (u,v)}^m\left(\genfrac{}{}{0ex}{}{m}{i}\right)p^i(1-p{)}^{m-i}\le \theta$$

$$\theta (W)=\underset{u\in W}{\max }\prod _{v\in W\setminus \{u\}}\sum _{i=\tau (u,v)}^m\left(\genfrac{}{}{0ex}{}{m}{i}\right)p^i(1-p{)}^{m-i}.$$

$$\displaystyle p=\frac{1}{m}\sum_{q\in\mathcal{Q}}\textstyle\bigl{|}\bigl{\{}\{u,v\}\in\binom{\mathcal{W}}{2}\,:\,l_{uq}=l_{vq}\bigr{\}}\bigr{|}\big{/}\binom{n}{2},$$

$$\underset{u\in W}{\min }\sum _{v\in W\setminus \{u\}}\gamma (u,v)$$

$$\mathrm{Pr}\left[\left|X-\mu \right|\ge \delta \right]\le 2e^{-2m{\delta }^2}.$$

$$r-\frac{\rho }{\left(\genfrac{}{}{0ex}{}{n}{2}\right)}$$

$$\frac{\rho }{\left(\genfrac{}{}{0ex}{}{n}{2}\right)}-1/s=\frac{n_{\mathrm{ex}}^2}{\left(\genfrac{}{}{0ex}{}{n}{2}\right)}\cdot \left(r-1/s\right)=\frac{n_{\mathrm{ex}}^2}{\left(\genfrac{}{}{0ex}{}{n}{2}\right)}\cdot \frac{s}{s-1}{\left(p_{\mathrm{ex}}-1/s\right)}^2>\frac{4(p_{\mathrm{ex}}-1/s{)}^2}{\left(\genfrac{}{}{0ex}{}{n}{2}\right)}\ge 2\delta .$$

$$\gamma (u,v)$$

$$=-\log \left(1-\sum _{i=0}^{\lceil m(\frac{1}{s}+\frac{\delta }{2})\rceil -1}\left(\genfrac{}{}{0ex}{}{m}{i}\right)(\frac{1}{s}+\delta {)}^i(1-\frac{1}{s}-\delta {)}^{m-i}\right)$$

$$\le -\log \left(1-2e^{-{\delta }^{2}m/2}\right)$$

$$-(n-1)\log \left(1-2e^{-{\delta }^{2}m/2}\right)$$

$$=-\log \left(2e^{-{\delta }^{2}m/2}+1-2n{e}^{-{\delta }^{2}m/2}\right)$$

$$\le -\log \left(2e^{-{\delta }^{2}m/2}\right).$$

$$\mathrm{Pr}[\overline{E_1}]\le \mathrm{Pr}\left[\left|\frac{{\sum }_{q\in \mathcal{Q}}{\kappa }_q}{m}-\rho \right|\ge \delta \cdot \left(\genfrac{}{}{0ex}{}{n}{2}\right)\right]\le 2e^{-2m{\delta }^2}.$$

$$\mathrm{Pr}\left[E_1\wedge \underset{(u,v)\in \left(\genfrac{}{}{0ex}{}{\mathcal{E}}{2}\right)}{\bigwedge }{E}_2(u,v)\wedge \underset{u\in \mathcal{W}\setminus \mathcal{E},v\in \mathcal{W}\setminus \{u\}}{\bigwedge }{E}_3(u,v)\right]$$

$$\le 1-2e^{-2m{\delta }^2}-n(n-1)e^{-m{\delta }^2/2}$$

$$\le 1-n^2\cdot e^{-m{\delta }^2/2}\le 1-ϵ.\text{qed}$$

$$\mathrm{Pr}\left[I_{a_q}\wedge \underset{c\in \mathcal{C}\setminus \{a_q\}}{\bigwedge }\overline{I_c}\right]$$

$$\ge 1-2e^{-2n\cdot {\left(\frac{p_{\mathrm{ex}}-1/s}{2}\right)}^2}-2(s-1)e^{-2n\cdot {\left(\frac{p_{\mathrm{ex}}-1/s}{2}\right)}^2}$$

$$=1-2s{e}^{-2n\cdot {\left(\frac{p_{\mathrm{ex}}-1/s}{2}\right)}^2}\ge 1-ϵ.$$

$$\mathrm{Pr}\left[{\mathtt{MV}}_q=a_q\right]$$

$$\le \frac{1}{2}+\sum _{i=n/2-n_{\mathrm{ex}}}^{n_{\mathrm{non}}/2}\left(\genfrac{}{}{0ex}{}{n_{\mathrm{non}}}{i}\right){\left(\frac{1}{2}\right)}^{n_{\mathrm{non}}}\le \frac{1}{2}+\frac{n_{\mathrm{ex}}}{2}\cdot \left(\genfrac{}{}{0ex}{}{n_{\mathrm{non}}}{n_{\mathrm{non}}/2}\right){\left(\frac{1}{2}\right)}^{n_{\mathrm{non}}}$$

$$\le \frac{1}{2}+\frac{n_{\mathrm{ex}}}{2}\cdot \frac{2^{n_{\mathrm{non}}}}{\sqrt{\pi n_{\mathrm{non}}/2}}{\left(\frac{1}{2}\right)}^{n_{\mathrm{non}}}=\frac{1}{2}+\frac{n_{\mathrm{ex}}}{\sqrt{2\pi n_{\mathrm{non}}}}\le \frac{1}{2}+ϵ.\text{qed}$$

Full text

1 Introduction

Crowdsourcing, which has become popular in recent years, is a process that requires completion of specific tasks by crowd workers. In crowdsourced single-answer multiple-choice questions, workers are asked to select one answer out of multiple candidates for each given question. Such a scheme is used, for instance, in annotating named entities for microblogs [12], sentiment classification on political blogs [15], and image tagging [24].

For the purpose of quality control, crowdsourcing systems usually assign multiple workers to the same questions and then aggregate their answers using some rule or algorithm. A critical fact here is that the workers often have different levels of expertise, skills, and motivation; some workers may guess the correct answers with their rich knowledge, while others may answer almost randomly to get a reward without any effort. Let us call the workers of the former and latter type experts and non-experts, respectively.

When only a few experts are overwhelmed by a large number of non-experts, the most intuitive answer aggregation rule, majority voting, often fails to acquire the correct answers. Consider a small example in Table 1. There are six workers $w_{1},\dots,w_{6}$ assigned to eight questions $q_{1},\dots,q_{8}$. For each question, the candidate answers are A, B, C, D, and E. Among these workers, $w_{1}$, $w_{2}$, and $w_{3}$ are experts who almost always give the correct answers, while the others, $w_{4}$, $w_{5}$, and $w_{6}$, are non-experts who give random answers. Let us apply the majority voting to their answers. The majority answer for $q_{1}$ is D, which is the correct answer. However, the majority answer for $q_{8}$ is C, which is not the correct answer. In addition, we need tie-breaking for $q_{3}$ because B and D get the same number of votes. It is very likely that as the fraction of non-experts increases, the quality of the majority voting answers deteriorates.

Various answer aggregation algorithms exist in addition to the majority voting (see Related Work). However, most of these algorithms implicitly strengthen the majority answers and therefore fail to provide the true answers when the majority answers are incorrect. To overcome this issue, Li et al. [21] recently proposed a sophisticated answer aggregation algorithm for such a hard situation. More specifically, they introduced the notion of hyper questions, each of which is a set of single questions. Their algorithm applies the majority voting (or other existing answer aggregation algorithms) for the hyper questions and then decodes the results to votes on individual questions. Finally, it applies the majority voting again to obtain the final answers. The results of their experiments demonstrate that their algorithm outperforms existing algorithms.

1.1 Our Contribution

In this study, we further investigate the above-mentioned hard situation. Our contribution can be summarized as follows:

We introduce a graph-mining-based efficient algorithm that accurately extracts the set of experts; 2. 2.

We propose two types of answer aggregation algorithms based on the above experts extraction algorithm; 3. 3.

We provide a theoretical justification of our proposed algorithms; 4. 4.

We conduct thorough computational experiments using synthetic and real-world datasets to evaluate the performance of our proposed algorithms.

First result.

To design a powerful answer aggregation algorithm for the above hard situation, it is crucial to extract reliable experts from the crowd workers. In the example above, if we recognize that $w_{1}$, $w_{2}$, and $w_{3}$ are experts, we can obtain the correct answers for all questions by simply applying the majority voting to the answers of the experts. The fundamental observation we use is as follows: as the experts almost always give the correct answers, each pair of experts frequently gives the same answer to a question. Let us now consider constructing an edge-weighted complete undirected graph in which each vertex corresponds to a worker and each edge weight represents the agreement rate of the answers of two workers. From the above observation, it is very likely that there is a dense component consisting of the experts, which we call the expert core. Note that the formal definition of the expert core will be given in Section 2. Figure 1 depicts an edge-weighted graph constructed from the example in Table 1. As can be seen, experts $w_{1}$, $w_{2}$, and $w_{3}$ form a dense component. Although, in this example, we simply set the edge weight to the number of same answers of two workers, we will use more suitable values in our algorithm. To extract the expert core, we use a well-known dense subgraph extraction algorithm called the peeling algorithm [2], which removes the most unreliable worker one by one.

Second result.

Based on the expert core extraction algorithm, we propose two types of answer aggregation algorithms. The first one incorporates the expert core into existing answer aggregation algorithms such as the majority voting. Indeed, once we extract the expert core, we can apply existing algorithms to the answers of the workers in the expert core. The second one utilizes the information pertaining to the reliability of workers provided by the expert core extraction algorithm, which is quite effective when the task is very hard.

Third result.

We first demonstrate that the expert core is very unlikely to contain a non-expert if the number of questions is sufficiently large (Theorem 2). We then prove that the majority voting is asymptotically correct if there are only experts (Theorem 3), but not reliable if there are a large number of non-experts (Theorem 4). Theorems 2 and 3 provide a theoretical justification for the first type of our algorithm. In fact, combining these two theorems, we see that if the number of questions and the number of workers in the expert core are sufficiently large, our proposed algorithm (i.e., the majority voting among the workers in the expert core) gives the correct answer with high probability. On the other hand, Theorem 4 provides the limitation of the majority voting, which implies that it is quite important to exclude non-experts when we use the majority voting.

Fourth result.

We conduct thorough computational experiments using synthetic and real-world datasets to evaluate our proposed algorithms. To simulate the hard situation in this study, we use six datasets recently collected by Li et al. [21] as real-world datasets, all of which have difficult heterogeneous-answer questions that require specialized knowledge. We demonstrate that the expert core counterparts of existing answer aggregation algorithms perform much better than their original versions. Furthermore, we show that our novel algorithm based on the information of the reliability of workers outperforms the other algorithms particularly when the task is quite hard.

1.2 Related Work

To date, a large body of work has been devoted to developing algorithms that estimate the quality of workers [8, 37, 36, 31, 16, 35, 4, 9, 19, 26, 27, 39, 3, 23, 34, 11, 28, 21, 38, 22]. In most existing work, the quality of workers and correct answers are estimated by an iterative approach comprising the following two steps: (i) infer the correct answers based on the quality of workers estimated and (ii) estimate the quality of workers based on the correct answers inferred. For example, Whitehill et al. [37] modeled each worker’s ability and each task’s difficulty, and then designed a probabilistic approach, which they called GLAD (Generative model of Labels, Abilities, and Difficulties).

Finding a dense component in a graph is a fundamental and well-studied task in graph mining. A typical application of dense subgraph extraction is to identify components that have some special role or possess important functions in the underlying system represented by a graph. Examples include communities or spam link farms extraction in Web graphs [10, 13], identification of molecular complexes in protein–protein interaction graphs [5], and expert team formation in collaboration graphs [6, 33], The peeling algorithm [2] is known to be effective in various optimization problems for dense subgraph extraction (e.g., the densest subgraph problem and its variations [1, 7, 17, 29, 18] as well as other related problems [33, 30]).

2 Model

An instance of our problem is a tuple $(\mathcal{W},\mathcal{Q},\mathcal{C},\mathcal{L})$, where each component is defined as follows: There is a finite set of workers $\mathcal{W}=\{w_{1},\ldots,w_{n}\}$ and a finite set of questions $\mathcal{Q}=\{q_{1},\ldots,q_{m}\}$. Each question $q$ has a set of candidate answers $\mathcal{C}_{q}=\{c^{q}_{1},\dots,c^{q}_{s}\}$ ($s\geq 2$). Suppose that, for each question $q$, worker $w$ answers $l_{wq}\in\mathcal{C}_{q}$ and let $\mathcal{L}=(l_{wq})_{w\in\mathcal{W},\,q\in\mathcal{Q}}$. Our task is to estimate the unknown correct answers to the questions.

Suppose that, among the workers $\mathcal{W}=\{w_{1},\dots,w_{n}\}$, there are $n_{\mathrm{ex}}$ experts $\mathcal{E}~{}(\subseteq\mathcal{W})$ who give the correct answer with probability $p_{\mathrm{ex}}~{}(>1/s)$, and the other $n_{\mathrm{non}}~{}(=n-n_{\mathrm{ex}})$ workers are non-experts who give an answer independently and uniformly at random. If an expert makes a mistake, she selects a wrong answer independently and uniformly at random. Thus, for a question $q\in\mathcal{Q}$ with a correct answer $a_{q}\in\mathcal{C}_{q}$ and an answer $c\in\mathcal{C}_{q}$, it holds that

[TABLE]

It should be noted that, by showing the candidate answers in random order for each worker, we can handle some biases (e.g., some non-expert workers always choose the first candidate of answers) using this model.

Let us consider the probability that a pair of workers $u,v\in\mathcal{W}$ $(u\neq v)$ gives the same answer for a question $q\in\mathcal{Q}$. If at least one of $u$ and $v$ is a non-expert, then we have $\Pr[l_{uq}=l_{vq}]=1/s$. On the other hand, if both workers are experts, then $\Pr[l_{uq}=l_{vq}]=\frac{(p_{\mathrm{ex}}-1/s)^{2}}{1-1/s}+\frac{1}{s}$, which is strictly larger than $1/s$.

For each pair of workers $u,v\in\mathcal{W}$, let $\tau(u,v)$ be the number of questions such that $u$ and $v$ give the same answer, that is, $\tau(u,v)=|\{q\in\mathcal{Q}\,:\,l_{uq}=l_{vq}\}|$. Here, if $\Pr[l_{uq}=l_{vq}]=p$, then $u$ and $v$ give the same answers for at least $\tau(u,v)$ questions with probability $\sum_{i=\tau(u,v)}^{m}\binom{m}{i}p^{i}(1-p)^{m-i}$.

For a given $p\in(0,1)$, a subset of workers $W\subseteq\mathcal{W}$ is called a $\theta$-expert set with respect to $p$ if

[TABLE]

holds for all $u\in W$. Intuitively, a $\theta$-expert set with small $\theta$ is a set of workers that is very unlikely to contain a non-expert. Let $\theta(W)$ be the minimum threshold such that $W$ is a $\theta$-expert set, that is,

[TABLE]

Then, we employ $W\subseteq\mathcal{W}$ that minimizes $\theta(W)$ as the estimated set of experts. We refer to such a set as the expert core (with respect to $p$). As will be shown in Theorem 2, the expert core is very unlikely to contain a non-expert if the number of questions is sufficiently large.

3 Algorithms

In this section, we design an algorithm to compute the expert core, and then propose two types of answer aggregation algorithms. Our expert core extraction algorithm first constructs an edge-weighted complete undirected graph that represents the similarity of the workers in terms of their answers. Then, it extracts a dense component in the graph using the peeling algorithm [2].

3.1 Peeling Algorithm

We first revisit the peeling algorithm. The algorithm iteratively removes a vertex with the minimum weighted degree in the current graph until we are left with only one vertex. Let $G=(V,E,\omega)$ be an edge-weighted graph. For $S\subseteq V$ and $v\in S$, let $d_{S}(v)$ denote the weighted degree of $v$ in the induced subgraph $G[S]$, that is, $d_{S}(v)=\sum_{e=\{u,v\}\in E:\,u\in S}\omega(e)$. Then, the procedure can be summarized in Algorithm 1. Note that the algorithm here returns $S_{i}\in\{S_{1},\dots,S_{|V|}\}$ that maximizes $d_{S_{i}}(v_{i})$, although there are other variants depending on the problem at hand [7, 17, 30, 33]. Algorithm 1 can be implemented to run in $O(|E|+|V|\log|V|)$ time.

Algorithm 1 indeed produces the $k$-core decomposition of graphs. For $G=(V,E,\omega)$ and a positive real $k$, a subset $S\subseteq V$ is called a $k$-core if $S$ is a maximal subset in which every vertex $v\in S$ has a weighted degree of at least $k$ in the induced subgraph $G[S]$. Note that a $k$-core is unique for a fixed $k$. The $k$-core decomposition reveals the hierarchical structure of $k$-cores in a graph and is particularly focused on finding the $k$-core with maximum $k$. Algorithm 1 is suitable for this scenario; in fact, it is evident that the algorithm returns the $k$-core with maximum $k$.

3.2 Expert Core Extraction Algorithm

Here, we present an algorithm to compute the expert core. In particular, we explain the construction of an edge-weighted graph, which represents the similarity of the workers in terms of their answers.

In our algorithm, we set $p$ to the average agreement probability, that is,

[TABLE]

where \binom{\mathcal{W}}{2}=\bigl{\{}\{u,v\}\,:\,u,v\in\mathcal{W},\ u\neq v\bigr{\}}, and extract the expert core with respect to this $p$ via the peeling algorithm. We construct a complete graph $(\mathcal{W},\binom{\mathcal{W}}{2})$ with weight $\gamma(u,v)=-\log\sum_{i=\tau(u,v)}^{m}\binom{m}{i}p^{i}(1-p)^{m-i}$, where recall that $\tau(u,v)=|\{q\in\mathcal{Q}\,:\,l_{uq}=l_{vq}\}|$. Then, we compute the $k$-core with maximum $k$ for the edge-weighted graph $(\mathcal{W},\binom{\mathcal{W}}{2},\gamma)$ using Algorithm 1. As a result, we can obtain a set of workers $W\subseteq\mathcal{W}$ such that the following value is maximized:

[TABLE]

As $-\log x$ is monotone decreasing, the obtained set minimizes $\theta(W)$ and hence is the expert core.

Note that we can construct the edge-weighted graph $(\mathcal{W},\binom{\mathcal{W}}{2},\gamma)$ in $O(n^{2}m)$ time and Algorithm 1 for the graph runs in $O(\binom{n}{2}+n\log n)=O(n^{2})$ time. Thus, we have the following theorem.

Theorem 1**.**

The expert core can be found in $O(n^{2}m)$ time.

3.3 Answer Aggregation Algorithms

Once we extract the expert core, we can apply existing answer aggregation algorithms (e.g., the majority voting) to the answers of the workers in the expert core, which implies the expert core counterparts of the existing algorithms. In addition, we propose a novel answer aggregation algorithm. The peeling algorithm produces the ordering that represents the reliability of the workers. With this ordering, we can obtain a majority-voting-based algorithm as follows. At the beginning, we obtain the ordering of workers using the peeling algorithm for $(\mathcal{W},\binom{\mathcal{W}}{2},\gamma)$. Then we pick the most reliable pair of workers, that is, the pair left until the second last round of the peeling algorithm. For each question, if the two workers select the same answer, then we employ this answer. If the two workers select different answers, then we add the next most reliable worker one by one according to the ordering until two of them give the same answer. Our algorithm, which we call Top-2, is summarized in Algorithm 2. Note that the algorithm runs in $O(n^{2}m)$ time, which is the same as that of the expert core extraction algorithm.

4 Theoretical Results

In this section, we provide a theoretical justification for the first type of our proposed algorithm. The proofs can be found in Appendix 7. Suppose that each worker gives answers according to the model described in Section 2.

The following theorem states that the expert core does not contain any non-expert with high probability if the number of questions is sufficiently large.

Theorem 2**.**

Let $W^{*}\subseteq\mathcal{W}$ be the expert core. If $n_{\mathrm{ex}}\geq 2$ and $m\geq\frac{2n^{4}\log\frac{n^{2}}{\epsilon}}{(p_{\mathrm{ex}}-1/s)^{4}}$ for $\epsilon>0$, then we have $\Pr[W^{*}\subseteq\mathcal{E}]\geq 1-\epsilon$.

The next theorem states that the majority voting gives the correct answer with high probability if the number of workers is sufficiently large and all of them are experts. Let $\mathtt{MV}_{q}$ be the output of the majority voting for question $q\in\mathcal{Q}$.

Theorem 3**.**

If $n=n_{\mathrm{ex}}\geq\frac{2\log\frac{2s}{\epsilon}}{(p_{\mathrm{ex}}-1/s)^{2}}$ for $\epsilon>0$, then we have $\Pr\left[\mathtt{MV}_{q}=a_{q}\right]\geq 1-\epsilon$ $(\forall q\in\mathcal{Q})$.

Finally, the following theorem states that, the majority voting does not provide the correct answer with high probability if the number of non-experts is much larger than the number of experts.

Theorem 4**.**

If $n_{\mathrm{non}}\geq\frac{n_{\mathrm{ex}}^{2}}{2\pi\epsilon^{2}}$ for $\epsilon>0$, then we have $\Pr\left[\mathtt{MV}_{q}=a_{q}\right]\leq\frac{1}{2}+\epsilon$ $(\forall q\in\mathcal{Q})$.

It should be noted that when the number of candidate answers is $2$, even the random choice gives the correct answer with probability $1/2$. Thus, the above theorem indicates that the majority voting is no better than the random choice if the number of non-experts is much larger than the number of experts.

5 Experiments

In this section, we report the results of computational experiments. The objective of our experiments is to examine the performance of our proposed algorithms from various aspects using both synthetic and real-world datasets. The main component of our experiments comprises comparing our proposed answer aggregation algorithms with the following three existing algorithms: MV (the majority voting), GLAD [37], and Hyper-MV [21]. As our proposed algorithms, we employ the expert core counterparts of the above three algorithms, which we denote by Ex-MV, Ex-GLAD, and Ex-Hyper-MV, respectively, in addition to Top-2.

5.1 Datasets

As for synthetic datasets, we generate a variety of instances using the model described in Section 2. Recall the following five parameters: number of workers $n=|\mathcal{W}|$, number of questions $m=|\mathcal{Q}|$, number of candidate answers (for each question) $s$, number of ground-truth experts $n_{\mathrm{ex}}$, and the probability $p_{\mathrm{ex}}$ that an expert gives the correct answer. Throughout the experiments, we set $s=5$.

Table 2 summarizes six datasets that we use as real-world datasets. They were recently collected by Li et al. [21] using Lancers, a commercial crowdsourcing platform in Japan. Consistent with the hard situation being addressed here, these datasets have difficult heterogeneous-answer questions that require specialized knowledge. In fact, as shown later, the majority voting performs poorly on these datasets. Note that the classical datasets used in other previous work (e.g., Bluebird and Dog in image tagging [36, 39], Web in Web search relevance judging [39], Price in product price estimation [25], and RTE and Temp in textual entailment recognition [32]) usually have relatively easy homogeneous-answer questions, which are not within the scope of this study. During the collection of the above six datasets, all workers were asked to answer all questions. This may not be the usual assumption in crowdsourcing but is effective in the current hard situation. Indeed, if we can identify reliable experts by asking all workers to answer a small number of questions at an early stage, then it is possible to ask only the identified experts to answer to the remaining questions, which may reduce the overall cost. This scenario has been studied in previous work [21, 20].

5.2 Experts Extraction

To evaluate the performance of our expert core extraction algorithm in terms of extracting reliable experts, we conducted simulations using synthetic datasets. We set the number of workers as $n$ to $20$ and the number of experts as $n_{\mathrm{ex}}$ to $4$. In these settings, we generated two types of instances to simulate different situations. In the first type, to investigate the effect of the number of questions, we vary the number of questions $m$ from 5 to 50 (in increments of 5) under a fixed value of $p_{\mathrm{ex}}=0.8$. In the second type, to investigate the effect of the expertise of workers, we vary the probability $p_{\mathrm{ex}}$ from 0.5 to 1.0 (in increments of 0.05) under a fixed value of $m=25$.

As the sets of workers extracted by our peeling algorithm, we employ the Top-2 pair (i.e., a pair of workers left until the second last round of the peeling algorithm) and the expert core. To evaluate the quality of the set of workers at hand, we adopt the following two measures: the precision (number of ground-truth experts in the set divided by the size of the set) and the recall (number of ground-truth experts in the set divided by the number of all ground-truth experts).

The results are shown in Figure 2. Each point corresponds to an average over 100 data realizations. As can be seen, the precision of the expert core becomes better as the number of questions $m$ or the probability $p_{\mathrm{ex}}$ increases. Moreover, the expert core is robust in terms of the recall; in fact, even when $p_{\mathrm{ex}}$ is small (i.e., $p_{\mathrm{ex}}\leq 0.7$), the expert core achieves an average recall of $0.8$. The Top-2 pair achieves better precision in all parameter settings.

5.3 Ordering Defined by Peeling

The objective here is to demonstrate that the ordering of workers defined by the peeling algorithm for $(\mathcal{W},\binom{\mathcal{W}}{2},\gamma)$ reflects the accuracy rate of the workers. To this end, we perform the peeling algorithm for the real-world datasets. The results are shown in Figure 3. In the subfigures, the horizontal axis represents the ordering of workers defined by the peeling algorithm; specifically, the peeling algorithm has removed the workers from right to left. The vertical axis represents the accuracy rate of the workers. The bars corresponding to the workers contained in the expert core are colored black, while those corresponding to the other workers are colored gray.

As can be seen, workers with smaller order tend to have higher accuracy rates, In addition, the expert core contains almost all workers with significantly high accuracy rates. which demonstrates the reliability of our peeling algorithm. In particular, for Pokémon and Science, the ordering defined by the peeling algorithm correctly reproduces the ordering of the top five workers in terms of accuracy rates.

5.4 Comparison with Existing Algorithms

Using both synthetic and real-world datasets, we compare the performance of our proposed answer aggregation algorithms, that is, Top-2, Ex-MV, Ex-GLAD, and Ex-Hyper-MV, with existing algorithms, that is, MV, GLAD, and Hyper-MV. We first explain the details of GLAD and Hyper-MV. GLAD is a method that takes into account not only the worker expertise, denoted by $\alpha$, but also the difficulty of each question, denoted by $\beta$, We set $\alpha\sim\mathcal{N}(1,1)$ and $\beta\sim\mathcal{N}(1,1)$ as in Li et al. [21]. It is known that GLAD runs in $O(nmsT)$ time, where $T$ is the number of iterations to converge. Hyper-MV is a method that applies the majority voting to the hyper questions rather than the original individual questions. Because the number of possible hyper questions may be too large, Li et al. [21] suggested to apply the following random sampling procedure for $r$ times: (i) shuffle the order of all single questions uniformly at random to generate a permutation and (ii) from this permutation, pick every $k$ single questions from the beginning of the queue to generate hyper questions as long as they can be picked. Then, the overall time complexity of Hyper-MV is given by $O(rmn)$. We performed the sampling procedure with $k=5$ for $r=100$ times, as suggested by Li et al. [21].

Table 4 shows the results for synthetic datasets. We list the results for nine settings in which the probability $p_{\mathrm{ex}}$ varies in $\{0.7,0.8,0.9\}$ and $n_{\mathrm{ex}}$ varies in $\{2,4,6\}$. We set the number of workers $n$ to 100 and the number of questions $m$ to 50. As can be seen, each expert core counterpart achieves better performance than the original algorithm. In particular, Ex-MV significantly improves the performance of MV. Top-2 outperforms the other algorithms particularly when the problem is quite difficult, although Ex-MV performs better when the problem is relatively easy.

Table 4 summarizes the results for the six real-world datasets. As can be seen, for all datasets except IT, our proposed algorithms achieve the best performance. In fact, for almost all datasets, the performance of the existing algorithms is improved by using the expert core. Among our proposed algorithms, Top-2 provides the best performance; in particular, for English and Science, the accuracy rate of Top-2 is even higher than those of the other algorithms. It should be noted that, for English, Medicine, and Science, the accuracy rate of Top-2 is strictly higher than the best accuracy rate among workers (presented in Table 2), which emphasizes the power of answer aggregation in crowdsourcing. According to the trend observed in synthetic datasets, it is very likely that the high performance of Top-2 stems from the high fraction of non-experts in real-world datasets.

6 Conclusion

In this study, we addressed the answer aggregation task in crowdsourcing. Specifically, we focused on a hard situation wherein a few experts are overwhelmed by a large number of non-experts. To design powerful answer aggregation algorithms for such situations, we introduced the notion of expert core, which is a set of workers that is very unlikely to contain a non-expert. We then designed a graph-mining-based efficient algorithm that exactly computes the expert core. Based on the expert core extraction algorithm, we proposed two types of answer aggregation algorithms. The first one incorporates the expert core into existing answer aggregation algorithms. The second one utilizes the information provided by the expert core extraction algorithm pertaining to the reliability of workers. In particular, we provided a theoretical justification for the first type of algorithm. if the number of questions and the number of workers in the expert-core are sufficiently large, our proposed algorithm gives the correct answer with high probability. Computational experiments using synthetic and real-world datasets demonstrated that our proposed answer aggregation algorithms outperform state-of-the-art algorithms.

There are several directions for future research. Our model assumes that all experts give the correct answer with the same probability and all non-experts give an answer independently and uniformly at random. However, in reality, experts themselves may have different levels of expertise and non-experts may not be completely random. Although we have already confirmed that our proposed algorithms work well on real-world datasets, it is interesting to extend our model to such a more general scenario. Another direction is to generalize our model in a higher-level perspective. This study has focused on crowdsourced closed-ended questions, where workers can select an answer from candidates. On the other hand, there are often cases where we wish to handle crowdsourced open-ended questions, where workers have to answer without any candidates. We believe that our proposed algorithms may be applicable to this more general scenario by introducing a measure of similarity of answers (and thus similarity of workers).

Acknowledgments

This work was partially supported by JST ACT-I Grant Number JPMJPR17U79 and JSPS KAKENHI Grant Numbers 16K16005, 17H07357, 18J23034, and 19K20218.

7 Omitted proofs in Section 4

In this section, we give proofs for Theorems 2–4. To prove Theorems 2 and 3, we use the Hoeffding’s inequality.

Theorem 5** (Hoeffding’s inequality [14]).**

Let $X_{1},\dots,X_{m}$ be independent random variables such that $0\leq X_{i}\leq 1$ for all $i$. Let $X=\sum_{i=1}^{m}X_{i}/m$ and set $\mu=\mathbb{E}[X]$. Then, for all $\delta>0$, we have

[TABLE]

7.1 Proof of Theorem 2

Theorem 2**.**

Let $W^{*}\subseteq\mathcal{W}$ be the expert-core. If $n_{\mathrm{ex}}\geq 2$ and $m\geq\frac{2n^{4}\log\frac{n^{2}}{\epsilon}}{(p_{\mathrm{ex}}-1/s)^{4}}$ for $\epsilon>0$, then we have $\Pr[W^{*}\subseteq\mathcal{E}]\geq 1-\epsilon$.

Proof.

Let $r=p_{\mathrm{ex}}^{2}+(s-1)\cdot\left(\frac{1-p_{\mathrm{ex}}}{s-1}\right)^{2}~{}(=\frac{s}{s-1}(p_{\mathrm{ex}}-1/s)^{2}+1/s)$. Without loss of generality, we may assume $n>n_{\mathrm{ex}}\geq 2$ and $\epsilon\leq 1$. We use the union bound to bound the probability of bad events. Let $\delta=\frac{(p_{\mathrm{ex}}-1/s)^{2}}{n^{2}}$ and $\rho=(\binom{n}{2}-\binom{n_{\mathrm{ex}}}{2})\cdot\frac{1}{s}+\binom{n_{\mathrm{ex}}}{2}\cdot r$. It should be noted that $1/s+2\delta<\rho/\binom{n}{2}<r-2\delta$ because

[TABLE]

and

[TABLE]

We define some events for $u,v\in\mathcal{W}$ as follows:

(a)

$E_{1}$: $1/s+\delta<p<r-\delta$;

(b)

$E_{2}(u,v)$: $m(r-\delta/2)\leq\tau(u,v)\leq m(r+\delta/2)$;

(c)

$E_{3}(u,v)$: $\tau(u,v)\leq m(1/s+\delta/2)$.

Suppose that the events $E_{1}$, $E_{2}(u,v)$ occur for all $(u,v)\in\binom{\mathcal{E}}{2}$, and $E_{3}(u,v)$ for all $(u,v)\in\binom{\mathcal{W}}{2}\setminus\binom{\mathcal{E}}{2}$. Then, for $(u,v)\in\binom{\mathcal{E}}{2}$, we have

[TABLE]

by the Hoeffding’s inequality. Also, for $(u,v)\in\binom{\mathcal{W}}{2}\setminus\binom{\mathcal{E}}{2}$, we have

[TABLE]

by the Hoeffding’s inequality. Since $m=\frac{4\log(n^{2}/\epsilon)}{\delta^{2}}\geq\frac{2\log(2n)}{\delta^{2}}$, we have

[TABLE]

Consider the sets $S_{1},\dots,S_{n}$ computed in Algorithm 1. We have $d_{S_{i}}(v)\leq-(n-1)\log\left(1-2e^{-\delta^{2}m/2}\right)$ if $v\in S_{i}\setminus\mathcal{E}$ and $d_{S_{i}}(v)\geq-\log\left(2e^{-\delta^{2}m/2}\right)$ if $v\in S_{i}\cap\mathcal{E}$ and $|S_{i}\cap\mathcal{E}|\geq 2$. Hence, $W^{*}\subseteq\mathcal{E}$ holds.

The proof is completed by showing that the events $E_{1}$, $E_{2}(u,v)$ for $(u,v)\in\binom{\mathcal{E}}{2}$, and $E_{3}(u,v)$ for $(u,v)\in\binom{\mathcal{W}}{2}\setminus\binom{\mathcal{E}}{2}$ occur at the same time with probability at least $1-\epsilon$.

For each $q\in\mathcal{Q}$, let $\kappa_{q}=\left|\{(u,v)\in\binom{\mathcal{W}}{2}\,:\,l_{uq}=l_{vq}\}\right|$. Then we have $\mathbb{E}\left[\kappa_{q}\right]=\rho$. By the Hoeffding’s inequality, we obtain

[TABLE]

Also, for $\{u,v\}\in\binom{n}{2}$, we get $\Pr[\overline{E_{2}}]\leq 1-\Pr\left[\left|\tfrac{\tau(u,v)}{m}-r\right|\geq\tfrac{\delta}{2}\right]\leq 2e^{-\frac{m\delta^{2}}{2}}$ by the Hoeffding’s inequality since $\Pr[l_{uq}=l_{vq}]=r$. Similarly, for each non-expert $u\in\mathcal{W}\setminus\mathcal{E}$ and worker $v\in\mathcal{W}\setminus\{u\}$, we have $\Pr[\overline{E_{3}}]\leq\Pr\left[\left|\tfrac{\tau(u,v)}{m}-r\right|\geq\tfrac{\delta}{2}\right]\leq 2e^{-\frac{m\delta^{2}}{2}}$ by the Hoeffding’s inequality since $\Pr[l_{uq}=l_{vq}]=1/s$.

By the above claims, we have

[TABLE]

7.2 Proof of Theorem 3

Theorem 3.

If $n=n_{\mathrm{ex}}\geq\frac{2\log\frac{2s}{\epsilon}}{(p_{\mathrm{ex}}-1/s)^{2}}$ for $\epsilon>0$, then we have $\Pr\left[\mathtt{MV}_{q}=a_{q}\right]\geq 1-\epsilon$ $(\forall q\in\mathcal{Q})$.

Proof.

Fix $q\in\mathcal{Q}$ and let $\rho=\frac{p_{\mathrm{ex}}+1/s}{2}$. For each $c\in\mathcal{C}$, let $I_{c}$ be an event such that $|u\in\mathcal{W}\,:\,l_{uq}=c\}|\geq\rho\cdot n$. Note that, $\mathtt{MV}_{q}=a_{q}$ holds if $I_{a_{q}}\wedge\bigwedge_{c\in\mathcal{C}\setminus\{a_{q}\}}\overline{I_{c}}$ occurs.

By the Hoeffding’s inequality, we obtain

[TABLE]

Hence, the claim holds. ∎

7.3 Proof of Theorem 4

Theorem 4.

If $n_{\mathrm{non}}\geq\frac{n_{\mathrm{ex}}^{2}}{2\pi\epsilon^{2}}$ for $\epsilon>0$, then we have $\Pr\left[\mathtt{MV}_{q}=a_{q}\right]\leq\frac{1}{2}+\epsilon$ $(\forall q\in\mathcal{Q})$.

Proof.

Note that we have $\binom{2k}{k}=(2k!)/(k!)^{2}\leq 4^{k}/\sqrt{\pi k}$ by Stirling’s formula. Thus, we obtain

[TABLE]