MiSC: Mixed Strategies Crowdsourcing

TL;DR

MiSC is a versatile framework that combines traditional crowdsourcing quality evaluation with tensor completion methods, improving label aggregation accuracy through an iterative Tucker algorithm.

Contribution

This work introduces MiSC, a novel framework integrating conventional crowdsourcing and tensor completion, along with a new iterative Tucker label aggregation algorithm.

Findings

Outperforms state-of-the-art label aggregation methods

Demonstrates effectiveness through extensive experiments

Successfully integrates mixed strategies for crowdsourcing

Abstract

Popular crowdsourcing techniques mostly focus on evaluating workers' labeling quality before adjusting their weights during label aggregation. Recently, another cohort of models regard crowdsourced annotations as incomplete tensors and recover unfilled labels by tensor completion. However, mixed strategies of the two methodologies have never been comprehensively investigated, leaving them as rather independent approaches. In this work, we propose $\textit{MiSC}$ ($\textbf{Mi}$xed $\textbf{S}$trategies $\textbf{C}$rowdsourcing), a versatile framework integrating arbitrary conventional crowdsourcing and tensor completion techniques. In particular, we propose a novel iterative Tucker label aggregation algorithm that outperforms state-of-the-art methods in extensive experiments.

Tables

Table 1. Table 1: Estimation errors (%) of pure and mixed strategies on Web, BM, RTE, Gog, Temp, and Bluebirds datasets. Nonzero rates and annotation error rates of datasets are given after their names ( ⋅ % / ⋅ % \cdot\%/\cdot\% ). As an example, the lowest estimation error in the Web dataset comes from the low-rank Tucker completion + MMCE (O) subscript MMCE (O) \text{MMCE}_{\text{(O)}} aggregation strategies.

Web (3.3/ 63.4)	MV	DS-EM	DS-MF	${\text{MMCE}}_{\text{(C)}}$	${\text{MMCE}}_{\text{(O)}}$	BM (6.0/ 31.1)	MV	DS-EM	DS-MF	${\text{MMCE}}_{\text{(C)}}$	${\text{MMCE}}_{\text{O)}}$
pure	$26.93$	$16.92$	$16.10$	$11.12$	$10.33$	pure	$30.4$	$27.60$	$26.90$	$27.10$	$27.10$
LRTC	$26.76$	$16.55$	$16.09$	$11.12$	$10.33$	LRTC	$29.25$	$27.60$	$26.90$	$27.10$	$27.10$
TenALS	$26.93$	$16.77$	$15.83$	$11.12$	$10.33$	TenALS	$27.60$	$27.60$	$26.90$	$27.10$	$27.10$
Tucker	$10.87$	$5.77$	$5.73$	$6.97$	$5.24$	Tucker	$26.50$	$27.00$	$26.20$	$26.40$	$26.40$
RTE (6.1/ 16.3)	MV	DS-EM	DS-MF	${\text{MMCE}}_{\text{(C)}}$	${\text{MMCE}}_{\text{(O)}}$	Dog (9.2/ 30.0)	MV	DS-EM	DS-MF	${\text{MMCE}}_{\text{(C)}}$	${\text{MMCE}}_{\text{(O)}}$
pure	$10.31$	$7.25$	$7.13$	$7.50$	$7.50$	pure	$17.78$	$15.86$	$15.61$	$16.23$	$16.73$
LRTC	$9.25$	$7.25$	$7.00$	$7.50$	$7.50$	LRTC	$15.61$	$15.61$	$15.61$	$15.61$	$15.61$
TenALS	$10.25$	$7.25$	$7.13$	$7.50$	$7.50$	TenALS	$15.86$	$15.74$	$15.61$	$15.86$	$15.86$
Tucker	$8.38$	$6.88$	$6.75$	$7.50$	$7.50$	Tucker	$15.61$	$15.49$	$15.37$	$15.86$	$15.86$
Temp (13.2/ 15.9)	MV	DS-EM	DS-MF	${\text{MMCE}}_{\text{(C)}}$	${\text{MMCE}}_{\text{(O)}}$	Bluebirds (100.0/ 36.4)	MV	DS-EM	DS-MF	${\text{MMCE}}_{\text{(C)}}$	${\text{MMCE}}_{\text{(O)}}$
pure	$6.39$	$5.84$	$5.84$	$5.63$	$5.63$	pure	$24.07$	$10.19$	$10.19$	$8.33$	$8.33$
LRTC	$5.19$	$5.63$	$5.63$	$5.63$	$5.63$	LRTC	$20.37$	$9.26$	$9.26$	$6.48$	$6.48$
TenALS	$5.41$	$5.63$	$5.84$	$5.63$	$5.63$	TenALS	$23.15$	$9.26$	$9.26$	$6.48$	$6.48$
Tucker	$5.19$	$4.98$	$4.98$	$5.41$	$5.41$	Tucker	$19.91$	$8.33$	$9.26$	$4.63$	$4.63$

Equations

$$\mathcal{A}(r_1,\cdots ,r_{k-1},j,r_{k+1},\cdots ,r_d)$$

$$\sum _{r_k=1}^{R_k}\mathbf{U}(j,r_k)\mathcal{G}(r_1,\cdots ,r_{k-1},r_k,$$

$$\mathcal{A}$$

$$=\mathcal{G}{\times }_1{\mathbf{U}}^{(1)}{\times }_2{\mathbf{U}}^{(2)}\cdots {\times }_d{\mathbf{U}}^{(d)},$$

$$=[[\mathcal{G};{\mathbf{U}}^{(1)},{\mathbf{U}}^{(2)},\dots ,{\mathbf{U}}^{(d)}]],$$

$$\mathcal{A}=\sum _{r=1}^R\mathcal{G}(r,\dots ,r)({\mathbf{u}}_r^{(1)}\circ \cdots \circ {\mathbf{u}}_r^{(d)}).$$

$$\mathbf{A}=\left(\begin{array}{ccc}1& 0& 4\\ 1& 3& 0\end{array}\right).$$

$$\mathcal{A}(1,:,:)=\left(\begin{array}{cccc}1& 0& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 1\end{array}\right),\mathcal{A}(2,:,:)=\left(\begin{array}{cccc}1& 0& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 0\end{array}\right).$$

$$\underset{\mathcal{G},{\mathbf{U}}^{(1)},{\mathbf{U}}^{(2)},{\mathbf{U}}^{(3)}}{\min }||[[\mathcal{G};{\mathbf{U}}^{(1)},{\mathbf{U}}^{(2)},{\mathbf{U}}^{(3)}]]-\mathcal{A}||,$$

$${\text{rank}}_{\text{Tucker}}([[\mathcal{G};{\mathbf{U}}^{(1)},{\mathbf{U}}^{(2)},{\mathbf{U}}^{(3)}]])=(R_1,R_2,R_3).$$

$$\mathbf{A}=\left(\begin{array}{ccc}1& 3& 4\\ 1& 3& 4\end{array}\right).$$

$$\mathcal{A}(1,:,:)=\mathcal{A}(2,:,:)=\left(\begin{array}{cccc}1& 0& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right).$$

Full text

1 Introduction

In recent years, with the advent of many complex machine learning models, the need for large amounts of labeled data is boosted. Though we can obtain unlabeled data abundantly and cheaply, acquiring labeled data from domain experts or well-trained workers with specific background knowledge is usually expensive and time-consuming. Crowdsourcing provides an effective way of collecting labeled data quickly and inexpensively. However, as crowd workers are usually non-experts or even spammers, the individual contribution from one worker can be unreliable. To improve the label quality, each item is presented to multiple workers. Therefore, inferring the true labels from a large sum of noisy labels is a key challenge in crowdsourcing.

Existing learning-from-crowds works mainly focus on eliminating annotations Li and Jiang (2018) from unreliable workers or reducing their weights. In line with this goal, a series of work focusing on evaluating workers’ quality are proposed based on accuracy Whitehill et al. (2009); Karger et al. (2011), and confusion matrix Dawid and Skene (1979); Raykar et al. (2009); Liu et al. (2012); Zhou et al. (2012); Zhang et al. (2016). Apart from that, two emerged works Zhou and He (2016) and Li and Jiang (2018) aim at completing annotations by translating single/multi-label crowdsouring tasks to tensor completion problems. However, their improvements in performance over conventional annotations methodologies are limited. In this work, we present a label aggregation algorithm by mixing the two strategies: We view the label tensor as an incomplete and noisy tensor, which is coherent to the reality where incompleteness comes from heavy workloads and noise stems from mislabelling. In a nutshell, we capture the structural information and filter out noisy information from the label tensor through tensor completion. This is followed by a conventional label aggregation procedure. We iterate over the above two steps until convergence. Despite the generality of the proposed framework, we highlight our use of tensor decomposition techniques.

Tensors are a higher-order generalization of vectors and matrices and constitute a natural representation for many real-life data that are intrinsically multi-way. In analogy to the significance of matrix QR factorization and singular value decomposition in matrix preconditioning and principal component analysis, tensor decomposition concepts have been deployed in modern machine learning models. Iconic examples include text analysis Collins and Cohen (2012); Liu et al. (2015), compression of neural networks Denton et al. (2014); Novikov et al. (2015), and tensor completion Suzuki (2015); Imaizumi et al. (2017). Among various popular tensor decomposition techniques, we specifically have interests in the Tucker model Tucker (1963); Levin (1963); De Lathauwer et al. (2000a). The Tucker decomposition factorizes a tensor into a core tensor of generally smaller size, along with factor matrices for each of the tensor modes. We exemplify through a case study and show that for a perfectly labeled matrix, its corresponding label tensor will have an intrinsically low-rank Tucker decomposition. The main contributions of this article are:

• •

A general mixed strategies framework for crowdsourcing tasks is proposed, in which annotations are completed by tensor recovery and ground-truth labels are estimated by conventional deduction algorithms.

• •

To our knowledge, this is the first work that introduces tensor decomposition methods to exploit the structural information in the label tensor. This scheme also bears a clear physical interpretation.

• •

Experimental results demonstrate that the proposed algorithms manage to improve existing methods to achieve higher aggregation accuracy.

• •

Most importantly, the proposed algorithms are shown to be particularly powerful compared to conventional pure label aggregation methods when the annotations are highly sparse and severely noisy. This is crucial since obtaining low-sparsity/ high-quality annotations can be arduous and expensive.

In the following, Section 2 briefly summarizes the related work. Then, Section 3 presents the necessary tensor preliminaries. Section 4 focuses on one of the proposed models, mixed low-rank Tucker DS-EM aggregation, and showcases the algorithm. Next, numerical experiments comparing the proposed MiSC (mixed strategies crowdsourcing) with pure label aggregation methods are given in Section 5. Finally, Section 6 concludes this work.

2 Related Work

Our two-stage mixed algorithms build upon the standard label aggregation algorithms. Majority voting, as the most straightforward crowdsourcing technique, has served as a baseline method for years. Besides, the seminal work of Dawid and Skene based on expectation maximization (EM), denoted as DS-EM henceforth, is also among the earliest work of crowdsourcing Dawid and Skene (1979) which is a generative method by assuming the performance of each worker is consistent across different tasks. Moreover, the crowdsourcing problem is further translated into a variational Bayesian inference problem of a graphical model in Liu et al. (2012). Using Mean Field algorithm, the parameters are tuned to maximize the marginal likelihood.  Zhou et al. (2012) introduces the minimax entropy principle into crowdsourcing tasks and infers true labels by minimizing the Kullback–Leibler (KL) divergence between the probability distribution over workers, items, labels, and the ground truth.

In matrix completion, most problems are formulated into the construction of a structurally low-rank matrix $\bm{X}$ having the same observed entries: $\min_{\bm{X}}{\textrm{rank}(\bm{X})},\textrm{ {s.t.} }(\bm{X}-\bm{A})_{\Omega}=\bm{0}$, where $\bm{A}$ represents the matrix with missing values filled by zeros and $\Omega$ is the mapping that specifies the locations of originally non-zero elements. Directly solving the above optimization problem is NP-hard, which results in extensive research on solving alternative formulations. One of the two popular candidates is to minimize the nuclear norm as the convex envelope of the matrix rank-operator Candès and Recht (2009); Chen (2015). This nuclear norm minimization idea is then further generalized to tensor completion problems by computing matricizations of the tensor along its $k$ modes and summing over the nuclear norm of the resulting $k$ matrices (abbreviated as LRTC in the remainder of this paper) Liu et al. (2013); Signoretto et al. (2014). Methods that exploit tensor decomposition formats were also introduced to tensor completion problems in recent years. In Jain and Oh (2014); Suzuki (2015); Zhao et al. (2015), the authors use the Canonical Polyadic (CP) decomposition for tensor estimators.

The pioneers of introducing tensor completion concepts to crowdsourcing problems are Zhou and He (2016) and Li and Jiang (2018), where authors build label aggregation algorithms upon the methodologies introduced in Liu et al. (2013). This work contrasts with them in two main aspects: (1) Aims - we focus on introducing a versatile complete-aggregate two-step looping structure for crowdsourcing tasks, where completion techniques adopted can be of any kind. (2) Approaches - we introduce tensor decomposition driven completion algorithms and showcase its advantages over LRTC methods in crowdsourcing tasks.

3 Preliminaries

Tensors are high-dimensional arrays that generalize vectors and matrices. In this paper, boldface capital calligraphic letters $\bm{\mathcal{A}},\bm{\mathcal{B}},\ldots$ are used to denote tensors, boldface capital letters $\bm{A},\bm{B},\ldots$ denote matrices, boldface letters $\bm{a},\bm{b},\ldots$ denote vectors, and Roman letters $a,b,\ldots$ denote scalars. A $d$-way or $d$-order tensor $\bm{\mathcal{A}}\in\mathbb{R}^{I_{1}\times I_{2}\times\cdots\times I_{d}}$ is an array where each entry is indexed by $d$ indices $i_{1},i_{2},\ldots,i_{d}$. We use the convention $1\leq i_{k}\leq I_{k}$ for $k=1,\ldots,d$, where the $I_{k}$ is called the $k$th dimension of tensor $\bm{\mathcal{A}}$. MATLAB notation $\bm{\mathcal{A}}(i_{1},i_{2},\ldots,i_{d})$ is used to denote entries of tensors. Fibers are high-dimensional analogue of rows and columns in matrices. In a matrix $\bm{A}$, a matrix column can be referred by fixing a column index. By again adopting MATLAB convention, the $i_{2}$th column of matrix $\bm{A}$ is denoted $\bm{A}(:,i_{2})$, which is also called a $1$-mode fiber of the matrix. Similarly, a matrix row $\bm{A}(i_{1},:)$ is known as a $2$-mode fiber of the matrix. For a $d$-way tensor, a $k$-mode fiber is determined by fixing all the indices but the $k$th one, which we denote as $\bm{\mathcal{A}}(i_{1},\ldots,i_{k-1},:,i_{k+1},\ldots,i_{d})$.

Tensor $k$-mode matricization, also known as tensor $k$-mode unfolding/flattening, reorders the elements of a $d$-way tensor into a matrix. It is formally defined as:

Definition 1

(Kolda and Bader, 2009, p. 459)* The $k$-mode matricization of a tensor $\bm{\mathcal{A}}\in\mathbb{R}^{I_{1}\times\cdots\times I_{d}}$, denoted by $\bm{A}_{(k)}\in\mathbb{R}^{I_{k}\times I_{1}\cdots I_{k-1}I_{k+1}\cdots I_{d}}$, arranges the $k$-mode fibers to be the columns of the resulting matrix. Tensor element $\bm{\mathcal{A}}(i_{1},\ldots,i_{d})$ maps to matrix element $\bm{A}_{(k)}(i_{k},j)$, where $j=1+\sum^{d}_{p=1,p\neq k}(i_{p}-1)J_{p}$ and $J_{p}=\prod^{p-1}_{m=1,m\neq k}I_{m}$.*

The generalization of the matrix-matrix multiplication to tensor involves a multiplication of a matrix with a $d$-way tensor along one of its $d$ modes:

Definition 2

(Kolda and Bader, 2009, p. 460)* The $k$-mode product of a tensor $\bm{\mathcal{G}}\in\mathbb{R}^{R_{1}\times\cdots\times R_{d}}$ with a matrix $\bm{U}\in\mathbb{R}^{J\times R_{k}}$ is denoted $\bm{\mathcal{A}}=\bm{\mathcal{G}}\,{\times_{k}}\,\bm{U}$ and defined by*

[TABLE]

where $\bm{\mathcal{A}}\in\mathbb{R}^{R_{1}\times\cdots\times R_{k-1}\times J\times R_{k+1}\times\cdots\times R_{d}}$.

Following Definition 2, the full multilinear product of a $d$-way tensor with $d$ matrices is:

Definition 3

*(*Cichocki et al., 2015, p. 147) The full multilinear product of a tensor $\bm{\mathcal{G}}\in\mathbb{R}^{R_{1}\times\cdots\times R_{d}}$ with matrices $\bm{U}^{(1)},\bm{U}^{(2)},\ldots,\bm{U}^{(d)}$, where $\bm{U}^{(k)}\in\mathbb{R}^{I_{k}\times R_{k}}$, is denoted $\bm{\mathcal{A}}=[[\bm{\mathcal{G}};\bm{U}^{(1)},\bm{U}^{(2)},\ldots,\bm{U}^{(d)}]]$ and defined by $\bm{\mathcal{A}}=\bm{\mathcal{G}}\,{\times_{1}}\,\bm{U}^{(1)}\,{\times_{2}}\,\bm{U}^{(2)}\cdots{\times_{d}}\,\bm{U}^{(d)}$, where $\bm{\mathcal{A}}\in\mathbb{R}^{I_{1}\times\cdots\times I_{d}}$.

Following the definition of the full multilinear product, we define the Tucker decomposition as follows:

Definition 4

The Tucker decomposition decomposes a tensor $\bm{\mathcal{A}}\in\mathbb{R}^{I_{1}\times\cdots\times I_{d}}$ into a core tensor $\bm{\mathcal{G}}\in\mathbb{R}^{R_{1}\times\cdots\times R_{d}}$ multiplied by a matrix $\bm{U}^{(k)}\in\mathbb{R}^{I_{k}\times R_{k}}$ along the $k$th mode for $k=1,\ldots,d$. That is, a $d$-way tensor $\bm{\mathcal{A}}$ is regarded as a multilinear transformation of a small core tensor $\bm{\mathcal{G}}$ by $d$ factor matrices $\bm{U}^{(1)},\bm{U}^{(2)},\ldots,\bm{U}^{(d)}$. By writing out $\bm{U}^{(k)}=[\bm{u}_{1}^{(k)},\bm{u}_{2}^{(k)},\ldots,\bm{u}_{R_{k}}^{(k)}]$ for $k=1,2,\dots,d$, we have

[TABLE]

where $r_{1},r_{2},\ldots,r_{d}$ are auxiliary indices that are summed over, and $\circ$ denotes the outer product.

The dimensions $(R_{1},R_{2},\ldots,R_{d})$ of these auxiliary indices are called the Tucker ranks. It is worth noting that $R_{k}$ is in general no bigger than $rank(\bm{A}_{(k)})$, which is also called the multilinear rank. In other words, for a Tucker representation with Tucker ranks $(S_{1},S_{2},\ldots,S_{d})$, if there exists a $k$, where $1\leq k\leq d$ and $S_{k}>rank(\bm{A}_{(k)})$, then we can always find an equivalent Tucker representation with Tucker ranks no bigger than the multilinear ranks.

When the core tensor $\bm{\mathcal{S}}$ is cubical and diagonal, a Tucker model reduces to a CP model. By writing out the formula, a CP decomposition expresses a $d$-way tensor $\bm{\mathcal{A}}$ as

[TABLE]

4 When Label Aggregation Meets Tensor Completion

We now exemplify the proposed complete-aggregate two-step looping scheme using the case of mixed low-rank Tucker DS-EM aggregation. Firstly, we showcase in Section 4.1 that Tucker model bears a clear physical meaning in completing label tensor. In Section 4.2, we explain in detail how to combine a low-rank Tensor completion with traditional label aggregation to form one loop. The MiSC algorithms are then given in 4.3.

4.1 Basic Idea

We use $\bm{A}\in\mathbb{R}^{N_{w}\times N_{i}}$ to denote the label matrix, where $N_{w}$ is the number of workers and $N_{i}$ is the number of items. If item $i$ is labeled by worker $w$ as class $c$, then we have $\bm{A}(w,i)=c$. Now we proposed to form a three-way binary tensor $\bm{\mathcal{A}}$ of size $N_{w}\times N_{i}\times N_{c}$ by using the label matrix. This is done by using the canonical basis vectors to denote the non-zero elements in the matrix and all-zero vectors to denote unlabeled entries. Due to the fact that each worker gives at most one label for one item, there then contains at most one $1$ in each of the $3$-mode fibers of tensor $\bm{\mathcal{A}}$. We use a small example to illustrate this process.

Example 1

Suppose there are two workers labeling for three items among four classes. The first worker believes the first item and the third item belong to the first class and the fourth class, respectively. The second worker labels the first item as the first class and the second item as the third class. We use label matrix $\bm{A}\in\mathbb{R}^{2\times 3}$ to describe the above setting, where

[TABLE]

Considering matrix $\bm{A}$ contains four non-zero elements, four $3$-mode fibers of the resulting tensor contain a $1$, while the remaining two fibers are initiated with zeros vectors. For $\bm{A}(1,1)=1$ and $\bm{A}(2,1)=1$, the canonical basis vector $\bm{e}_{1}=(1,0,0,0)^{T}$ is used. For $\bm{A}(1,3)=4$ and $\bm{A}(2,2)=3$, the canonical basis vectors $\bm{e}_{4}=(0,0,0,1)^{T}$ and $\bm{e}_{3}=(0,0,1,0)^{T}$ are needed respectively. Collecting these fibers renders the following tensor

[TABLE]

We call the above-defined tensor $\bm{\mathcal{A}}$ label tensor. Our goal is to complete the label tensor $\bm{\mathcal{A}}$ by assuming a specific underlying structure. Hence the problem can be regarded as a tensor completion task. As is well-known, the quality of the label matrix is in general not guaranteed. Therefore, it is of great importance that the proposed method should not only fill in entries but also serve as a pre-processor to de-noise the label tensor before doing deduction.

Similar to the traditional matrix completion problem formulations, in this paper we propose to form the following optimization problem:

[TABLE]

The optimization problem is solved via a truncated Tucker decomposition. It is worth noting that finding an exact Tucker decomposition of rank $(R_{1},R_{2},R_{3})$, where $R_{k}=rank(\bm{A}_{(k)})$, of $\bm{\mathcal{A}}$ is easy. However, finding an optimal truncated Tucker decomposition is nontrivial Kolda and Bader (2009). In this paper, the Tucker model is initialized using a truncated higher-order SVD De Lathauwer et al. (2000a) and updated by a higher-order orthogonal iteration algorithm De Lathauwer et al. (2000b). We now use an extreme case to demonstrate the vivid physical meaning behind the choice of a Tucker model.

Example 2

Suppose the label matrix is perfectly given by two workers for three items among four classes as follows

[TABLE]

The label tensor is then constructed as

[TABLE]

Now we consider the exact Tucker decomposition of this label tensor. By taking three different mode matricizations, we obtain $rank(\bm{\mathcal{A}}_{(1)})=1,rank(\bm{\mathcal{A}}_{(2)})=3,rank(\bm{\mathcal{A}}_{(3)})=3$, where $\min{(N_{i},N_{c})}=\min{(3,4)}=3$.

As is demonstrated in the above example, if given a noise-less label tensor, it can be naturally decomposed into an exact low-rank Tucker format with Tucker ranks equal to $(1,\min{(N_{i},N_{c})},\min{(N_{i},N_{c})})$.

4.2 Methodology

In general, the proposed MiSC consists of two phases: label tensor completion and deduction. In each phase, a wide range of algorithms are available as will be seen in Section 5. We visualize the architecture of MiSC in Figure 2.

The method starts from a binarized label tensor constructed as described in Section 4.1. A conventional label aggregation algorithm is adopted to infer a $1\times N_{i}$ resultant initial guess of the ground-truth labels. Similar to the idea explained in Example 1, the initial guess can then be encoded to a $1\times N_{i}\times N_{c}$ slice $\bm{\mathcal{S}}$ and concatenated with the raw binarized label tensor. By doing so, we enlarge the size of the label tensor from $N_{w}\times N_{i}\times N_{c}$ to $(N_{w}+1)\times N_{i}\times N_{c}$ and form a target tensor $\bm{\mathcal{T}}=[\bm{\mathcal{A}};\bm{\mathcal{S}}]$, which follows by completion iterations. The all-zero $3$-mode fibers of tensor $\bm{\mathcal{A}}$ are filled in during the annotation completion step as depicted in (2) and (3). At the same time, the imposed low-rank constraint also automatically smooths the noisy label tensor. When the stopping criteria are not satisfied, the bottom annotated slice of the completed tensor is separated and concatenated again with the binarized raw data to form an iterative refinement loop.

4.3 Algorithms

Before proposing the pseudocode of the MiSC algorithms, we will briefly introduce the truncated higher-order SVD (HOSVD) procedure De Lathauwer et al. (2000a) (denoted as Algorithm 1) and the higher-order orthogonal iteration (HOOI) algorithm De Lathauwer et al. (2000b) (denoted as Algorithm 2), which serve as two cornerstones for the low-rank Tucker completion.

The ultimate goal of Algorithms 1 and 2 is to find a (nearly) optimal low-rank Tucker decomposition approximation of the target tensor $\bm{\mathcal{T}}$. While finding an exact Tucker decomposition of the multilinear ranks is rather straightforward by directly employing HOSVD algorithm, the Tucker decomposition of ranks smaller than the multilinear ranks found by truncated HOSVD is demonstrated to be non-optimal in Kolda and Bader (2009), in terms of the norm of the difference. Hence we only use a truncated HOSVD step as an initialization procedure. Algorithm 1 takes in a tensor $\bm{\mathcal{T}}$ and prescribed Tucker ranks as inputs, and followed by consecutive SVDs operating on different mode matricizations. Factor matrices of the Tucker model are determined by the leading left-singular vectors within each SVD step. Lastly, the core tensor $\bm{\mathcal{G}}$ is computed through the full multilinear product of the tensor $\bm{\mathcal{T}}\in\mathbb{R}^{I_{1}\times\cdots\times I_{d}}$ with matrices $\bm{U}^{(1)T},\bm{U}^{(2)T},\ldots,\bm{U}^{(d)T}$. An initial Tucker decomposition is thereby obtained. It is worth noting that in the HOSVD algorithm, the mode matricizations are always computed directly from the input tensor. Hence the computation of each of the factor matrices is completely independent of other factor matrices.

The higher-order orthogonal iteration (HOOI) algorithm was proposed in De Lathauwer et al. (2000b) to give a low-rank Tucker decomposition with a smaller norm of the difference, without guarantees to converge to the global optimum. In Algorithm 2, a Tucker decomposition of prescribed initial ranks is firstly constructed using Algorithm 1. Then the updates follow an alternating linear scheme (ALS): the factor matrices are updated iteratively by updating one at a time and keeping all the others fixed. In contrast to HOSVD, the update of a factor matrix in HOOI is not independent of others. Specifically, the $i$-mode matricization $\bm{A}_{(i)}$ in HOOI is computed from tensor $\bm{\mathcal{A}}$, where $\bm{\mathcal{A}}\in\mathbb{R}^{R_{1}\times\cdots\times R_{i-1}\times I_{i}\times R^{0}\times\cdots\times R^{0}}$ in the first ALS sweep (a full for-loop corresponds to one sweep). Starting from the second ALS sweep, the size of tensor $\bm{\mathcal{A}}$ before $i$-mode matricization becomes $R_{1}\times\cdots\times R_{i-1}\times I_{i}\times R_{i+1}\times\cdots\times R_{d}$. One can update the Tucker decomposition for a fixed amount of sweeps or until the residual stops decreasing.

We summarize the proposed MiSC algorithms with an exemplary Tucker completion case in Algorithm 3. The for-loop realizes the label matrix to label tensor conversion process explained in Example 1. Within the while-loop, HOOI subroutines are employed to fill in and augment the label tensor. One can iterate over the filling process for a prescribed amount of sweeps or until the last slice stops evolving. After the loops, we binarize the completed 3-way tensor which then follows by a deduction step. The binarization is realized by choosing the indices of the maximum values in each of the $3$-mode vectors. A new $(N_{w}+1)\times N_{i}$ label matrix $\hat{\bm{A}}$ is thereby constructed. The most computationally expensive steps in Algorithm 3 are SVDs in the HOSVD and HOOI subroutines, which have computational complexities of approximately $O((I_{1}I_{2}I_{3})^{2}/I_{i})$ and $O(I_{i}(R^{0})^{4})$ flops, respectively.

5 Experimental Results

In this section, the proposed mixed complete-aggregate strategies crowdsourcing algorithms are compared with conventional label aggregation methods on six popular datasets, including Web dataset Zhou et al. (2012), BM dataset Mozafari et al. (2014), RTE dataset Snow et al. (2008), Dog dataset Deng et al. (2009); Zhou et al. (2012), Temp dataset Snow et al. (2008), and Bluebirds dataset Welinder et al. (2010). Details of their nonzero rates defined by $\frac{\text{\#worker~{}labels}}{\text{\#items}\times\text{\#workers}}$, and annotation error rates defined by $\frac{\text{\#wrong~{}labels}}{\text{\#worker~{}labels}}$ are recorded in Table 1.

Five conventional crowdsourcing methods are considered in the aggregation step of the proposed MiSC. They are: Majority Voting (MV), Dawid-Skene model + Expectation Maximization (DS-EM), Dawid-Skene model + mean field (DS-MF) Liu et al. (2012), and Categorical/ Ordinal Minimax Conditional Entropy ($\text{MMCE}_{\text{(C)}}$, $\text{MMCE}_{\text{(O)}}$).

We consider three tensor completion algorithms in the MiSC algorithms: LRTC111http://www.cs.rochester.edu/u/jliu/code/TensorCompletion.zip, TenALS222http://web.engr.illinois.edu/~swoh/software/optspace, and Tucker. These methods represent three different approaches towards the completion problem. LRTC Liu et al. (2013) aims at minimizing the sum of nuclear norms of the unfolded matrices. While TenALS Jain and Oh (2014) and Tucker utilize CP decomposition and Tucker factorization, respectively.

5.1 Comparison with State-of-the-Arts

The estimation errors of all five pure conventional label aggregation algorithms and fifteen MiSC approaches on six real-life datasets are reported in Table 1. It is shown that the proposed MiSC algorithms achieve lower estimation errors than traditional pure crowdsourcing methods on all six datasets. We observe the greatest breakthrough in the Web dataset, where the state-of-the-art estimation errors of around $10.33\%$ are brought down to around $5.24\%$. The second noticeable refinement is in the Bluebirds dataset, where MiSC algorithms produce error rates lower than $5\%$, in contrast to the $>8\%$ state-of-the-art records. For the BM, RTE, Dog datasets, the MiSC algorithm via DS-MF+Tucker also reaches the lowest errors of $26.2\%$, $6.75\%$, and $15.37\%$, respectively, among others.

5.2 Pure vs. Mixed Strategies Crowdsourcing

Besides evaluating all MiSC algorithms together with pure label aggregation methods, we also zoom in on pairwise comparisons between pure and mixed strategies that use the same label deduction approaches. Specifically, in the Web dataset, the MiSC algorithms stacked by DS-EM/ DS-MF and Tucker completion have estimation errors of $5.77\%/5.73\%$, compared with the initial $16.92\%/16.10\%$ by pure DS-EM/ DS-MF. In the Temp dataset, both DS-EM+Tucker and DS-MF+Tucker improve the performance of their corresponding pure counterparts by approximately one point. In the Bluebirds dataset, mixing MMCE aggregations with Tucker completion helps the crowdsourcing reach the lowest error rate of $4.63\%$ compared with the original $8.33\%$.

Consequently, we have empirically verified that the proposed MiSC algorithms can consistently improve the performance on top of conventional label aggregation schemes. By and large, we also remark that this complete-aggregate two-step looping structure is readily compatible with other state-of-the-art label deduction and tensor completion algorithms.

5.3 MiSC for Sparse and Noisy Annotations

As noticed in Section 5.1, although MiSC successfully improves the annotation quality in all six datasets, the striking refinement obtained in Web dataset stands out and raises the question: when will MiSC be remarkably advantageous? Referencing Table 1, one can see that the Web dataset has the sparsest and “poorest” annotations. Only $88$ items out of $2665$ are labeled on average per worker, and $63.4\%$ of the total $15567$ labels are misleading. We emulate similar scenarios by simultaneously eliminating worker labels from and adding noise to the RTE dataset. The resultant label matrix has a nonzero rate of $3.7\%$, and four different degrees of annotations error rates: $30.4\%,41.6\%,51.4\%,60.9\%$. Figure 3 records the corresponding error rates. We can then see that MiSC is significantly more robust to high sparsity and severe noise. Replacing reliable annotations by erroneous labels only slightly affects the accuracy of MiSC, while the performance of traditional pure label aggregation degrades rapidly. We highlight this important finding since the number and quality of worker labels have a huge implication on the crowdsourcing cost. To this end, the proposed MiSC approach provides a means to substantially relax the labor and labeling quality requirements while maintaining superior accuracy.

6 Conclusion

This paper has introduced the mixed strategies crowdsourcing (MiSC) framework, a versatile complete-aggregate two-step iterative procedure, for crowdsourcing tasks. MiSC is readily compatible with various completion techniques and deduction schemes. By integrating tensor completion procedures, and importantly, tensor decomposition methods, into label aggregation, the proposed methods can largely improve the crowdsourcing performance. By further assuming a low-rank Tucker structure, the mixed low-rank Tucker model with conventional label aggregation approaches are shown to be particularly favorable when the annotations are highly sparse and severely noisy. Experiments have shown that MiSC consistently outperforms state-of-the-art methods in terms of estimation errors.

Acknowledgment

This work is partially supported by the General Research Fund (Project 17246416) of the Hong Kong Research Grants Council.