Latent Distribution Assumption for Unbiased and Consistent Consensus Modelling

TL;DR

This paper proposes a novel latent distribution assumption for aggregating noisy labels, allowing for multiple subjective labels per object, which improves modeling in ambiguous labeling scenarios.

Contribution

It introduces a new latent distribution assumption that accounts for subjective label variability, enhancing consensus modeling for ambiguous data.

Findings

Better performance on difficult, ambiguous tasks

More suitable for scenarios with label uncertainty

Improves unbiased and consistent consensus estimation

Abstract

We study the problem of aggregation noisy labels. Usually, it is solved by proposing a stochastic model for the process of generating noisy labels and then estimating the model parameters using the observed noisy labels. A traditional assumption underlying previously introduced generative models is that each object has one latent true label. In contrast, we introduce a novel latent distribution assumption, implying that a unique true label for an object might not exist, but rather each object might have a specific distribution generating a latent subjective label each time the object is observed. Our experiments showed that the novel assumption is more suitable for difficult tasks, when there is an ambiguity in choosing a "true" label for certain objects.

Tables

Table 1. Table 1 : Accuracy (in %) of various models across real data sets. Results for the six nontrivial models are shown relative to the RFE result. The best results for each data set are shown in bold.

		Dawid & Skene		GLAD		Minimax entropy
Data set (number of classes)	RFE	LA	DA	LA	DA	LA	DA
Duchenne smiles ($K=2$)	72.08	+4.65	+4.03	+3.4	+5.28	-31.82	+0.88
Web search ($K=5$)	73.03	+12.69	+6.66	+8.2	+7.3	+3.04	+10.88
TREC ($K=3$)	45.48	+6.43	+3.47	-0.78	-0.1	+0.72	+1.51
Textual entailment ($K=2$)	89.92	+2.7	+2.33	+3.2	+2.08	-5.55	-2.42
Temporal ordering ($K=2$)	93.55	+0.61	+0.82	+0.39	+0.82	-2.21	-1.56
Adult content ($K=4$)	76.04	-0.36	+1.44	-0.36	-0.96	-1.56	+2.34
Price($K=7$)	32.5	-1.25	+1.25	0.0	0.0	-2.5	+2.5

Table 2. Table 2 : Log loss results across real data sets. The best results for each data set are shown in bold.

		Dawid & Skene		GLAD		Minimax entropy
Data set (number of classes)	RFE	LA	DA	LA	DA	LA	DA
Duchenne smiles ($K=2$)	$\infty$	2.467	0.821	2.074	0.753	3.464	0.817
Web search ($K=5$)	$\infty$	0.247	0.364	0.408	0.468	0.747	0.331
TREC ($K=3$)	$\infty$	1.403	1.143	1.28	1.116	1.779	1.196
Textual entailment ($K=2$)	0.509	0.554	0.363	0.308	0.377	0.9	0.454
Temporal ordering ($K=2$)	$\infty$	0.841	0.26	0.331	0.252	0.817	0.286
Adult content ($K=4$)	$\infty$	1.345	0.507	0.875	0.535	1.385	0.501
Price ($K=7$)	$\infty$	80.365	3.128	16.357	0.917	42.651	1.675

Table 3. Table 4 : Summary statistics for real data sets.

Data set	# Cl.	# Obj.	# Workers	# Samples	Lab. per obj.	Lab. per worker		Gr. truth
					mean	mean	med.	objects
Duchenne smiles	2	2134	64	30319	14.2	473.7	109	159
Web search	5	2665	177	15567	5.8	87.9	19	2653
TREC	3	20026	762	91783	4.6	120.5	18	3275
Textual entailment	2	800	164	8000	10.0	48.8	20	800
Temporal ordering	2	462	76	4620	10.0	60.8	20	462
Adult content	4	11040	825	92721	8.4	112.4	18	1517
Price	7	80	155	12400	155.0	80.0	80	80

Equations

$$\displaystyle\small\sum_{j\in J}\log\bigg{[}\sum_{z_{j}\in\mathcal{Y}}\operatorname{Pr}(z_{j}\mid p)\prod_{w\in W_{j}}\operatorname{Pr}(y_{j}^{w}\mid z_{j})\bigg{]},$$

$$\displaystyle\small\sum_{j\in J}\log\int\operatorname{Pr}(q_{j}\mid p)\prod_{w\in W_{j}}\bigg{[}\sum_{z^{w}_{j}\in\mathcal{Y}}\operatorname{Pr}(z^{w}_{j}\mid q_{j})\operatorname{Pr}(y_{j}^{w}\mid z_{j}^{w})\bigg{]}\operatorname{d}q_{j}.$$

$$\displaystyle\small\sum_{j\in J}\sum_{w\in W_{j}}\log\bigg{[}\sum_{z^{w}_{j}\in\mathcal{Y}}\operatorname{Pr}(z^{w}_{j}\mid q_{j})\operatorname{Pr}(y_{j}^{w}\mid z_{j}^{w})\bigg{]}.$$

$$\operatorname{Pr}(Y_{j}^{w}=y\mid Z_{j}=z_{j}):=P_{\rm GLAD}(Y^{w}_{j}\mid Z_{j}=z_{j})=\left\{\begin{array}[]{ll}\!\!\!\!a(w,j),&\!\!\!\!{\rm for~{}}y=z_{j};\\ \!\!\!\!\frac{1-a(w,j)}{K-1},&\!\!\!\!{\rm for~{}}y\in\{1,\ldots,K\}\setminus z_{j}.\\ \end{array}\right.$$

$${\hat{q}}^{\mathrm{LA}}(n_1):=\frac{r{a}^{n_1}(1-a{)}^{n-n_1}}{r{a}^{n_1}(1-a{)}^{n-n_1}+(1-r)(1-a{)}^{n_1}{a}^{n-n_1}},$$

$${\mathrm{Pr}}^{\mathrm{DA}}(N_1=n_1\mid N=n):=\left(\genfrac{}{}{0ex}{}{n}{n_1}\right)[qa+(1-q)(1-a){]}^{n_1}[q(1-a)+(1-q)a{]}^{n-n_1}.$$

$$\operatorname{\mathds{E}}(\hat{q}^{\rm LA}):=\sum_{n_{1}=0}^{n}\hat{q}^{\rm LA}(n_{1})\operatorname{Pr}^{\rm DA}(N_{1}=n_{1}\mid N=n)\\ =\sum_{n_{1}=0}^{n}\Bigg{[}\frac{{n\choose n_{1}}[qa+(1-q)(1-a)]^{n_{1}}[q(1-a)+(1-q)a]^{n-n_{1}}}{ra^{n_{1}}(1-a)^{n-n_{1}}+(1-r)(1-a)^{n_{1}}a^{n-n_{1}}}ra^{n_{1}}(1-a)^{n-n_{1}}\Bigg{]}$$

$$\underset{n\to \infty }{\mathrm{plim}}\frac{n_1}{n}=qa+(1-q)(1-a).$$

$${\hat{q}}^{\mathrm{LA}}=\frac{1}{1+\frac{1-r}{r}{\left(\frac{a}{1-a}\right)}^{n\left(1-2n_1/n\right)}},$$

$$\operatorname*{plim}_{n\to\infty}\hat{q}^{\rm LA}=\operatorname*{plim}_{n\to\infty}\frac{1}{1+\frac{1-r}{r}\left[\left(\frac{a}{1-a}\right)^{\left(1-2(qa+(1-q)(1-a))\right)}\right]^{n}}=\left\{\begin{array}[]{ll}0,&{\rm if}~{}q<\frac{2a-1}{4a-2}=0.5;\\ r,&{\rm if}~{}q=0.5;\\ 1,&{\rm if}~{}q>0.5.\end{array}\right.$$

Full text

1 Introduction

Crowdsourcing marketplaces, such as Amazon Mechanical Turk111https://www.mturk.com, make it possible to label large data sets in a shorter time as well as at a lower cost comparing to that needed for a limited number of experts. However, as workers at the marketplaces are non-professional and vary in levels of expertise, such labels are much noisier than those obtained from experts. In order to reduce the noisiness, typically, each object is labelled by several workers, and then these labels are further aggregated in a certain way to infer a more reliable consensus label for the object. Most advanced consensus models [e.g., 5, 15, 18] address different aspects of uncertainty in the process of generating noisy labels.

A traditional setting used in those and other previous studies is based on the latent label assumption (L-assumption), implying that each object has a unique latent true label, and, when a worker observes the object, this latent true label is corrupted with regard to a chosen stochastic model into an observed noisy label. As a consequence, consensus models designed under this assumption explain any disagreements among observed noisy labels of an object by the mistakes made by some of the labellers. However, this may not explain a certain kind of disagreements among labels produced by experts, which is typical for some object domains. E.g., when assessing relevance of documents to search queries, even well trained experts may disagree about the true label for certain objects [see e.g., 14]. This is equivalent to saying that a unique true label of an object does not exist, but rather each object has its specific distribution over possible subjective labels, which is induced by a distribution of personal preferences over different aspects of the task. This type of uncertainty lies beyond the traditional L-assumption, and this paper introduces a novel approach based on the latent distribution assumption (D-assumption) to deal with this problem.

The novel D-assumption suggests the following generative process: Each object has its specific (latent) distribution over subjective values of label. Each time a worker observes the object, a latent subjective label is sampled from the object’s distribution. Then, this subjective label is corrupted according to a stochastic model, and an observed label is revealed. It is crucial that the object’s subjective label introduced in the process is generated each time a worker observes the object. This distinguishes the latent distribution approach from the previous consensus models.

Besides, to the best of our knowledge, this is the first paper that looks at the output probabilities of consensus labels from a statistical point of view. In particular, we notice that posterior probabilities of labels obtained under the traditional L-assumption are poor estimates of the underlying true probabilities. In contrast to this, we show, both analytically and experimentally, that the probabilities obtained under the D-assumption may serve as accurate statistical estimates of the underlying true probabilities. Thus, the latent distribution for subjective labels of an object estimated via our framework can be further used for the problems of label distribution learning considered in [6].

**Background. **A line of previous studies on consensus modelling [e.g., 5, 15] explicitly assumed that each object has a single “true” label. Other works [e.g., 16, 17, 18] including Bayesian approaches [e.g., 1, 8, 13], made the L-assumption implicitly: though it is assumed that each object is associated with a probabilistic label, it is not allowed that a latent label of an object takes multiple different values. In other words, posterior probabilities for latent labels, inferred under the traditional L-assumption, is a measure of confidence in each value being a unique latent true label. Our work is orthogonal to those studies as it considers a different assumption for the process of generating noisy labels and leads to essentially different and “better calibrated” output probabilities. [10] suggested a consensus model for subjective tasks, our work is more fundamental in the sense that we describe a general framework to consensus modelling in such tasks and theoretically study properties of their outputs.

2 Latent distribution assumption

Let $Z_{j}$ be a random variable whose value is the latent label for an object $j$, and $Y^{w}_{j}$ be a random variable whose value is the observed noisy label assigned by a worker $w$ to an object $j$.

To the best of our knowledge, all existing consensus models do not go beyond the general generative model shown in Figure 1(a). This class of models, including [5, 15, 16] and others, which we call the LA-model, represents the following generative process:

A unique latent true label $z_{j}$ for object $j$ is drawn from a probability distribution $P_{Z}$ with a parameter $p$ (note that $P_{Z}$ is the same for all objects). For multiclass labels, $P_{Z}$ is usually a multinomial distribution $\operatorname{Mult}(p)$, where $p$ is a vector of prior probabilities. 2. 2.

Given the latent true label $z_{j}$ for an object $j$, an observed noisy label $y_{j}^{w}$ from a worker $w$ for this object is sampled from the conditional probability distribution $\operatorname{Pr}(Y_{j}^{w}\mid Z_{j}=z_{j})$222As usual, capital letters are used for random variables and the corresponding lower case letters are for particular values of the random variables. with two potential parameters333“Potential parameters” means that some of the parameters may be absent. $e^{w}$ and $d_{j}$. Intuitively, $e^{w}$ represents a level of expertise for the worker $w$, and $d_{j}$ corresponds to a level of difficulty for the object $j$.

Note that most previous studies are focused on the second step. They proposed different ways to model the corruption of the true label and, for example, additionally employed latent communities of workers [13] or features of the object and the worker [11]. However, this does not put these models beyond the general two-steps scheme described above.

In this paper, by using the novel D-assumption we change the first step and introduce the novel class of models, which we call the DA-model. At this point, for each object $j$, we define a random variable $Q_{j}$ whose value $q_{j}$ is a parameter of the distribution of subjective labels for this object. The graphical model is shown in Figure 1(b) and represents the following generative process:

For each object $j$, a parameter $q_{j}$ is drawn form a probability distribution $P_{Q}$ parameterised by $p$ (again, note that $P_{Q}$ is the same for all objects). 2. 2.

When a worker $w$ observes an object $j$:

1. (a)

A subjective label $z^{w}_{j}$ is sampled from the object’s distribution with the parameter $q_{j}$. 2. (b)

A noisy label $y^{w}_{j}$ is generated from the conditional probability distribution $\operatorname{Pr}(Y_{j}^{w}\mid Z_{j}=z^{w}_{j})$ with two potential parameters $e^{w}$ and $d_{j}$.

Observe that the key difference between Figures 1(a) and 1(b) is in the workers’ plate: for the DA-model, the workers’ plate is extended over the latent subjective labels $z_{j}^{w}$. Next we show that this difference constitutes a conceptual change for the inference of consensus labels.

Consider a set of noisy labels $\{y^{w}_{j}\}$ for objects $j\in J$ from workers $w\in W$. Traditionally, consensus labels are inferred by maximising the log-likelihood of the observed noisy labels. For the L-assumption, the consensus output for object $j$ is probability $\operatorname{Pr}(Z_{j}=z_{j})$ for each possible value $z_{j}$, reflecting our confidence that the value $z_{j}$ is the unique true label for object $j$. In this setting, the log-likelihood of the observed data is

[TABLE]

where the second sum is taken over the set $\mathcal{Y}$ of possible label values, $W_{j}$ is the set of indices of workers’ who labelled $j$-th object, and notations of the form $\operatorname{Pr}(x)$ are used instead of $\operatorname{Pr}(X=x)$ for short, and for the same reason we omitted possible parameters $e^{w}$ and $d_{j}$ in $\operatorname{Pr}(y_{j}^{w}\mid z_{j})$. Whereas, under the D-assumption, the consensus output for object $j$ is the distribution of subjective labels $z^{w}_{j}$, parameterized by the object’s parameter $q_{j}$. And then the log-likelihood is defined as

[TABLE]

To make the discussion of DA-models easier, in the rest of this paper we assume that parameter $p$ of the DA-model is a deterministic distribution assigning to each object a chosen beforehand (but unknown) distribution of the object’s subjective labels. Then, we can remove the parameter $p$ in Figure 1(b) and assume that $q_{j}$ are additional parameters of the model, and the expression for the log-likelihood of the observed labels becomes the following:

[TABLE]

Remark. Two described approaches resemble, to some extent, two approaches to topic modelling – the mixture of unigrams model and the latent Dirichlet allocation model [see, e.g., 3]. The first model assumes that each document is associated with a single topic, which specifies the distribution of words in this document; for the second model, each document may be associated with multiple topics, and words in this document are generated from a mixture of the distributions for the topics. By replacing topics for latent labels and words for noisy labels, we get the two approaches discussed in this section. Note, however, that for our setting, it is necessary that the domains of latent and observed labels are the same. So that latent “topics” are in one-to-one correspondence with the label values. Otherwise, we may not interpret the latent distribution.

Finally, let us motivate the novel approach by the following numerical example. Consider 210 workers with highly confident estimates of their expertise inferred from their labels for many tasks they completed. Let each worker assigns the correct label with probability $a:=0.8$ (independent of the value of that correct label). Consider a binary classification task with uninformative prior over the two labels $\{0,1\}$. Let each of the workers provides one noisy label for a new object, and it turns out that 110 of the labels are 1s and 100 are 0s. Under the L-assumption, the posterior probability $p_{1}$ of the true label being 1 is $p_{1}\approx a^{10}/(a^{10}+(1-a)^{10})>1-2^{-20}$, i.e. the correct label is practically 1 with probability 1, and we observe a very unreliable event that 100 out of 210 workers made a mistake. On the contrast, under the D-assumption, according to (2), the probability $p_{1}$ should maximize $110\log(0.8p_{1}+0.2(1-p_{1}))+100\log(0.2p_{1}+0.8(1-p_{1}))$, and we infer that most of the workers provided true (though subjective) labels and for this object the latent Bernoulli distribution has the parameter $p_{1}\approx 0.54$, which seems much more realistic.

3 Consensus models under L- and D-assumptions

A distinguishing feature of the DA-model, shown in Figure 1(b), is that latent subjective labels for an object are sampled each time the object is observed. Thus, given a certain LA-model, the corresponding DA-one is defined as follows: Firstly, the distribution $P_{Q}$ in the DA-model is defined over the domain of the parameter $p$ in the LA-model. E.g., for the examples of models below, $p$ is the parameter of the multinomial distribution $P_{Z}$, and $P_{Q}$ is defined to be a Dirichlet distribution. Secondly, The conditional distribution $\operatorname{Pr}(Y_{j}^{w}\mid Z_{j}=z^{w}_{j})$ in the DA-model is the conditional distribution of noisy labels defined in the LA-model, with the subjective label value $z^{w}_{j}$ for the random variable $Z_{j}$.

The rest of this section describes three established models444For all the models, let labels take values from $\mathcal{Y}:=\{1,\ldots,K\}$. as special cases of the traditional LA-model, and their novel counterparts as special cases of the novel DA-model. Later, in Section 6, we will empirically compare the performance of all these models.

**3.1 Dawid and Skene model

**Consider a special case of the LA-model with parameters $p$ and $\{e^{w}\}$ defined as follows: $p:=(p_{1},\ldots,p_{K})$ is the vector of prior probabilities for label values; the parameter $e^{w}$ is the confusion matrix of size $K\times K$, and $\operatorname{Pr}(Y^{w}_{j}=y^{w}_{j}\mid Z_{j}=z_{j}):=e^{w}(z_{j},y^{w}_{j})$. The model was proposed in [5] and we will refer to it as LA DS.

The corresponding special case of the DA-model is the following: (1) for each object $j$, a vector $q_{j}=(q_{j,1},\ldots,q_{j,K})$ is drawn from a Dirichlet distribution $P_{Q}:=\operatorname{Dir}(p)$, this vector $q_{j}$ is the parameter of the multinomial distribution of subjective labels for this object; (2) when a worker $w$ observes an object $j$, first, a subjective label $z^{w}_{j}$ is drawn from the multinomial distribution $\operatorname{Mult}(q_{j})$, and then, a noisy label $y^{w}_{j}$ is drawn from the multinomial distribution $\operatorname{Pr}(Y_{j}^{w}\mid Z_{j}=z^{w}_{j}):=\operatorname{Mult}(e^{w}(z^{w}_{j},\cdot))$, where $e^{w}(k,\cdot)$ stands for $k$-th row of the confusion matrix $e^{w}$. It will be denoted as DA DS.

**3.2 GLAD

**Consider the following special case of the LA-model with parameters $\{e^{w}\}$, and $\{d_{j}\}$: parameter $p$ is absent meaning that true labels $z_{j}$ are assumed to be deterministic; the parameter $e^{w}$ is a scalar value representing the level of expertise for the worker $w$; and the parameter $d_{j}$ is a scalar value representing the level of difficulty for the object $j$. Let $a(w,j):=\frac{1}{1+\exp{(-e^{w}d_{j})}}$ and define

[TABLE]

This model is called GLAD555GLAD is a shortcut for Generative model of Labels, Abilities, and Difficulties. and was described in [15], it will be denoted as LA GLAD.

The corresponding DA-model is the following: (1) each object $j$ has a deterministic parameter $q_{j}\in[0,1]$; (2) for a worker $w$ and an object $j$, a latent subjective label $z^{w}_{j}$ is sampled form the multinomial distribution $\operatorname{Mult}(q_{j})$, and then, a noisy label $y^{w}_{j}$ is generated from $\operatorname{Pr}(Y_{j}^{w}\mid Z_{j}=z^{w}_{j}):=P_{\rm GLAD}(Y^{w}_{j}\mid Z_{j}=z^{w}_{j})$. We will refer to it as DA GLAD.

**3.3 Minimax entropy principle

**Consider a model described in [18, Section 2] for multiclass labels; this model will be referred to as LA MME. The model was derived using a minimax entropy principle, and it is a special case of the LA-model with $\{e^{w}\}$ and $\{d_{j}\}$ defined as matrices of size $K\times K$. Using these parameters, for each worker $w$ and each object $j$, define $\operatorname{Pr}(Y^{w}_{j}=y^{w}_{j}\mid Z_{j}=z_{j}):=P_{\rm MME}(Y^{w}_{j}\mid Z_{j}=z_{j})\propto\exp{\left(e^{w}(z,y)+d_{j}(z,y)\right)}$.

The DA-counterpart is the following: (1) each object $j$ has a deterministic parameter $q_{j}\in[0,1]$; (2) for a worker $w$ and an object $j$, a latent subjective label $z^{w}_{j}$ is sampled from the multinomial distribution $\operatorname{Mult}(q_{j})$, and then, a noisy label $y^{w}_{j}$ is generated from $\operatorname{Pr}(Y_{j}^{w}\mid Z_{j}=z^{w}_{j}):=P_{\rm MME}(Y^{w}_{j}\mid Z_{j}=z^{w}_{j})$. This will be denoted as DA MME.

4 Theoretical analysis

In this section, we analyse the ability of two approaches to consensus modeling (based on the L- and D-assumptions) to output “calibrated” probabilities of labels.

Consider one object, assume $\mathcal{Y}=\{0,1\}$. Consider the generative process for the DA-model as described in Section 2, where $z\sim\operatorname{Bernoulli}(q)$, and $q$ is an unknown object-specific parameter. Assume $n$ workers provide labels for the object independently, and each of them makes a mistake with a known probability $1-a$, which is the same for all workers.

L-assumption approach. According to the traditional approach, the unknown parameter $q$ is estimated by the posterior probability of label $1$ for the object: $\hat{q}^{\rm LA}(n_{1}):=\frac{ra^{n_{1}}(1-a)^{n-n_{1}}}{ra^{n_{1}}(1-a)^{n-n_{1}}+(1-r)(1-a)^{n_{1}}a^{n-n_{1}}},$ where $r$ is the prior probability of label $1$ (common for all objects) and $n_{1}$ is the number of ones among $n$ noisy labels.

Proposition 1. * The value of $\hat{q}^{\rm LA}$ is neither unbiased, nor consistent estimate of $q$.*

Proof. The generative process based on the D-assumption defines the following distribution of the number of ones $N_{1}$: $\operatorname{Pr}^{\rm DA}(N_{1}=n_{1})={n\choose n_{1}}[qa+(1-q)(1-a)]^{n_{1}}[q(1-a)+(1-q)a]^{n-n_{1}}$. Consider the expectation of $\hat{q}^{\rm LA}$ with respect to this distribution: $\operatorname{\mathds{E}}(\hat{q}^{\rm LA})=\sum_{n_{1}=0}^{n}\hat{q}^{\rm LA}(n_{1})\operatorname{Pr}^{\rm DA}(N_{1}=n_{1})$. One can see that this is a biased estimate for $q$ (see the details in Appendix A).

Now we check the convergence of $\hat{q}^{\rm LA}$ to $q$ as $n$ tends to infinity. By the law of large numbers, the fraction of ones among noisy labels converges: $\operatorname*{plim}_{n\to\infty}\frac{n_{1}}{n}=qa+(1-q)(1-a)$. Assume that workers are not malicious, i.e. $a>0.5$. Note that $\hat{q}^{\rm LA}=\frac{1}{1+\frac{1-r}{r}\left(\frac{a}{1-a}\right)^{n\left(1-2n_{1}/n\right)}}$, therefore $\hat{q}^{\rm LA}$ converges to different values depending on $q$ (see the details in Appendix A). Thus, $\hat{q}^{\rm LA}$ is not a consistent estimate for $q$ unless the distribution $\operatorname{Bernoulli}(q)$ is degenerate or $r=q=0.5$. $\square$

**D-assumption approach. ** According to the novel approach based on the D-assumption, $N_{1}$ is a binomial random variable with parameters $(n,f(q))$, where $f(q):=qa+(1-q)(1-a)$ is the probability that an observed noisy label equals $1$. The estimate of parameter $q$ is the value $\hat{q}$ providing the maximum for the log likelihood of the observed noisy labels: $\hat{q}^{\rm DA}(n_{1}):=\arg\max_{\hat{q}}[n_{1}\log(f(\hat{q}))+(n-n_{1})\log(1-f(\hat{q}))].$

Proposition 2. * The value of $\hat{q}^{\rm DA}$ is an unbiased and consistent estimate of $q$.*

Proof. Note that $\hat{q}^{\rm DA}:=f^{-1}(\hat{s})$, where $\hat{s}:=\frac{n_{1}}{n}$. Therefore, we have $\operatorname{\mathds{E}}\hat{q}^{\rm DA}=\operatorname{\mathds{E}}f^{-1}(\hat{s})=f^{-1}(\operatorname{\mathds{E}}\hat{s})=f^{-1}(\frac{1}{n}\operatorname{\mathds{E}}N_{1})=q.$ The second equality uses linearity of $f^{-1}$. Note that $\hat{s}=n_{1}/n$ is a consistent estimate of the success probability $f(q)$ of binomial $N_{1}$. Therefore, $\hat{q}^{\rm DA}=f^{-1}(\hat{s})$ is a consistent estimate of $f^{-1}(f(q))=q$. $\square$

5 Details of implementation

In this section we describe our implementation for the six consensus models described in Sections 3.1–3.3. Our implementation allows us to directly maximise the log-likelihood of observed noisy labels instead of using EM, which optimises the lower bound for the log-likelihood.

For LA-models the log-likelihood (1) is a function of parameters $\{e^{w}\}_{w\in W},\{d_{j}\}_{j\in J}$, and $p$. To maximise the log-likelihood we use a standard conjugate gradient descent algorithm. And to compute gradients at each iteration we use a proprietary library for the automatic differentiation, which is an implementation of the operator overloading approach [see, e.g., 2]. Finally, using the maximum likelihood estimates for the parameters, we compute the consensus output for each object $j$ as the posterior probabilities for all $z\in\mathcal{Y}$: $\operatorname{Pr}^{\rm LA}(Z_{j}=z)\propto\operatorname{Pr}(z\mid p)\prod_{w\in W_{j}}\operatorname{Pr}(y_{j}^{w}\mid z,e^{w},d_{j})$

For DA-models the log-likelihood (2) is a function of parameters $\{e^{w}\}_{w\in W},\{d_{j}\}_{j\in J}$, and $\{q_{j}\}_{j\in J}$ and the same approach is used to maximize it. The consensus output for each object $j$ is the estimated distribution of the object’s subjective labels, i.e. for all $z\in\mathcal{Y}$: $\operatorname{Pr}^{\rm DA}(Z_{j}=z)=\operatorname{Pr}(z\mid q_{j})$

The output of any consensus model depends on the initial values for parameters. However, it will not influence our comparisons of LA- and DA-versions of models: the same initial values are used for common parameters in each pair of models. For the pair of DS models, confusion matrices are initialised as follows: for each object $j$ we first compute the vector of relative frequencies $q^{\rm RFE}_{j}$ of noisy labels for the object, then for each worker $w$ we compute a matrix of counts by considering each label of the worker and adding the corresponding vector $q^{\rm RFE}_{j}$ to the column $y^{w}_{j}$ of the worker’s matrix of counts, finally we normalise each row in the worker’s matrix of counts to make it sum up to 1. Parameter $p$ for LA DS is initialised by the frequency of each label value among all noisy labels. For the pair of GLAD models, workers’ parameters are initialised by 1 and objects’ parameters are initialised by $\exp(1)$ as suggested in the original paper [15]. For the pair of MME models, workers’ confusion matrices are initialised by computing the matrices of counts for workers for each worker (in the same way as for DS models) and then taking the logarithm for each element of the matrices666To avoid zeros in the matrices we add 1 to each element in the matrices of counts for workers and objects., and objects’ parameters are initialised in a similar way: for each worker $w$ we compute $r^{\rm RFE}_{w}$ the vector of relative frequencies of noisy labels from this worker, then for each object $j$ we compute its matrix of counts by considering each noisy label for this object and adding the corresponding vector $r^{\rm RFE}_{w}$ to the row $y^{w}_{j}$ of the matrix of counts, finally we take the logarithm for each elements in the matrix777Such nontrivial initialisation for confusion matrices in DS and MME models leads to better results comparing to trivial uniform initialisation for all matrix elements.. Parameters of objects’ latent distributions, $\{q_{j}\}_{j\in J}$, in all DA-models are initialised by the frequency of noisy labels for each object $j$.

6 Experiments

In this section we empirically show that: (1) the novel D-assumption is more realistic than the L-one, and, as a result, DA-models produce more accurate consensus labels; (2) models designed for the D-assumption are capable for recovering latent distributions of labels. A straightforward way to check the first proposition, is to compare performance of an LA-model against the corresponding DA-counterpart. However, as distributions of labels for real objects are unknown, we use synthetic data sets to check the second statement.

Consider multiclass labels $\{1,\ldots,K\}$. For each object $j$, a model M produces a consensus label $\hat{q}^{\rm M}_{j}$, that is a vector of length $K$ and its $k$-th element $\hat{q}^{M}_{j}[k]$ is the probability for the label value $k$. Note that all models, LA- and DA-, produce probabilistic labels, but, as we explained in the previous sections, the intuition behind the LA-model probabilistic output is “confidence” in each label value, whereas for the DA-one it is estimated probability of labels.

Relative frequency estimator (RFE)

Given a set of noisy labels for an object $j$, a standard baseline for consensus modeling is to estimate probabilities for all $z\in\mathcal{Y}$ as $\operatorname{Pr}^{\rm RFE}(Z_{j}=z):=\frac{\left|\{w\in W_{j}:y^{w}_{j}=z\}\}\right|}{\left|W_{j}\right|}$, where $W_{j}$ is the set of indices of workers that label the object $j$.

For each data set, the training set is a set of objects with multiple noisy labels for each object, and the test set is a subset of those objects with a known ground truth label for each of them. The quality of consensus labels produced from the training set is evaluated over the test set. Let $T$ be a set of indices of the test set objects, and, for an object $i\in T$, $t_{i}$ be its ground truth label. We use the following two metrics to evaluate performance of a model on the test set:

*Accuracy. * For each object $j$ in a training set, a model M produces a probabilistic label $\hat{q}^{\rm M}$, and an estimated label for this object is $\hat{t}^{\rm M}_{j}:=\arg\max_{k}\hat{q}^{\rm M}_{j}[k]$.888If the maximum is attained at several label values, we randomly choose one of them. In our experiments, RFE accuracy is averaged over 10 runs to reduce the effect of random tie-breaking. There was no need to average results of the nontrivial models as no tie was observed. Accuracy over a test set is defined as ${\rm acc}:=\frac{1}{\left|T\right|}\sum_{j\in T}\mathds{I}(t_{j}=\hat{t}^{\rm M}_{j}),$ where $\mathds{I}(X)$ is the indicator function for $X$.

*Log loss. * Given consensus labels $\hat{q}_{j}^{\rm M}$ based on a model M, the log loss over a test set is the mean negative log likelihood of the test set labels, computed as ${\rm logloss}:=-\frac{1}{\left|T\right|}\sum_{j\in T}\log_{K}{\left(q^{\rm M}_{j}[t_{j}]\right)}$. 999For this definition, log loss for non-informative probabilistic labels, consisting of probabilities $\frac{1}{K}$, is 1.

Real-world data. First, compare the seven models across the following data sets: Duchenne smiles [15], Web search [18], TREC [4], Textual Entailment [12], Temporal Ordering [12], Adult content [7], Price [9]. For each data set, ground truth labels for a subset of objects are provided by the domain experts. (See Appendix B for the tasks description and summary statistics of the data sets.) Table 1 shows accuracy results. For a data set, accuracy of each nontrivial model is shown as the difference between the result for the RFE baseline (given in the second column) and the model result. As a summary, among 21 comparisons of LA- and DA-models, the DA-counterpart is better in 13 out of 21 times. Besides this, the novel approach generates the best model for 4 out of 7 data sets. Table 2 shows the log loss of the seven models 101010To avoid infinite losses for RFE we applied add-one smoothing to $q_{j}^{\rm RFE}$. across the six data set. Remarkably, for the log loss, among 21 comparisons of LA- and DA-models, the DA-counterpart is better in 18 out of 21 times. And for 5 out of 7 data sets, the novel approach generates the best model. Let us highlight some interesting observations from Tables 1 and 2. First, consider the Textual entailment data set: according to the log loss, performance of the RFE baseline is infinitely bad111111This happens if none of noisy labels for an object is equal to the ground truth. for all the data sets except the Textual entailment, which may indicate that the task was easy, in the sense that the traditional L-assumption is likely to be true, and the DA-models suffer due to their greater flexibility. Secondly, consider the TREC data set, where according to the log loss, the quality of probabilities obtained by any model, LA or DA, is worse than that for non-informative uniform probabilities. We checked, that the mean number of labels per object for this data set is $4.6$ which may be not enough to get realistic distributions over three classes. And similarly we explain the results for Web search data set with the mean number of labels per object is $5.8$.

The number of labels per object. To analyse how the number of labels per object affects the performance of the LA- and DA-models, we conducted the following experiment. As an example, we take the Duchenne smiles data set and two models, LA GLAD and DA GLAD. Figure 2 shows accuracy and the log loss for the Duchenne smiles data as the mean number of labels per object grows from $3$ to $14.2$. For each integer $n\ \in[3;20]$, we sampled a part of the data set, such that the number of labels per object not greater than $n$, it was done as following: given a number of noisy labels for an object, we randomly draw $n$ of those labels (if the number of noisy labels for the object was $\leq n$, all the noisy labels were taken). For the sample of the data set, we obtained consensus labels using the two models and evaluate their performance as before. Results shown in Figure 2 are averaged over $10$ simulations. Indeed, when the number of noisy labels is $3$, the DA-model is worse than the LA-one according to both performance metrics. However, as the number of labels per object increases, the performance of the DA-model notably exceeds that of the LA-model at almost all overlap values. Especially, for the log loss, we can see that for the DA-model, the bigger the overlap of noisy labels per object, the better the probabilistic labels. Whereas, for the LA-model, when the number of labels per object grows, aggregated probabilistic labels converge to deterministic values and the log loss for the test set becomes worse.

Synthetic data. Inspired by the analysis in Section 4, next we empirically examine the ability of the two approaches for consensus modelling to recover the underlying probabilities for labels. Data for this experiment is generated similarly to [15]: we used $N=2000$ objects and 20 workers, each worker labels each object; for each object we randomly generate its latent parameter $q_{j}\sim\operatorname{U}[0,1]$, which is the probability of label $1$; to produce noisy labels we generate the following parameters for GLAD $e^{w}\sim\operatorname{U}[0,4]$ and $\log(d_{j})\sim\operatorname{U}[0,3]$. Using the parameters, a data set is generated according to the DA GLAD. Then, for the data set, we used our implementations for the LA GLAD and DA GLAD to obtain estimates $\hat{q}_{j}$. Table 3 shows the mean squared error, $\frac{1}{N}\sum_{j=1}^{n}(\hat{q}_{j}-q_{j})^{2}$, averaged across 10 simulations. This confirms that DA-model is able to accurately estimate the underlying true probabilities. Figure 3 shows calibrations plot for estimated probabilities obtained using the two approaches to consensus modelling. It demonstrates that probabilities estimated by the DA-model are empirically well calibrated, whereas the LA-model is biased towards degenerate distributions.

7 Conclusion

We have proposed the novel D-assumption for consensus modelling and a way of constructing novel DA-models under the assumption. It has been shown that the DA-models are able to obtain proper estimates for true probabilities in contrast to those for the established LA-models. Our experiments confirm that the DA-models often perform better for real data sets, implying that, the D-assumption is more realistic than the L-one. However, there is no universal model which is the best for every data set (even among the state-of-the-art models). Thus, it always makes sense to produce golden labels (by professional judges) at least for a small part of any data set to evaluate all existing models and check which one serves the best for the data at hand. In that sense, our novel D-assumption approach is complimentary to the family of existing consensus models and allows to twice increase their number via considering our alternative noisy label generation process to be implemented for each one of them.

Appendix A Theoretical analysis for the latent label assumption

Remind, that we consider one object whose “true” label $z\sim\operatorname{Bernoulli}(q)$, where $q$ is an unknown object-specific parameter. Given $n$ noisy labels for this object, such that each noisy label is correct with a known probability $a$, the goal is to estimate the parameter $q$.

According to the LA-model, the unknown parameter $q$ is estimated by the posterior probability of label $1$ for the object:

[TABLE]

where $r$ is the prior probability of label $1$ (common for all objects) and $n_{1}$ is the value of the random variable $N_{1}$, which is the number of ones among $n$ noisy labels.

Proposition 1**.**

The posterior probability of label $1$ obtained under the L-assumption is neither unbiased, nor consistent estimate of the true probability $q$ underlying the generative process for the DA-model.

Proof.

The posterior probability of label $1$ for the object is

[TABLE]

where $r$ is the prior probability of label $1$ (common for all objects) and $n_{1}$ is the value of the random variable $N_{1}$, which is the number of ones among $n$ noisy labels.

The generative process based on the D-assumption defines the following distribution over the number of ones $n_{1}$:

[TABLE]

Consider the expectation of $\hat{q}^{\rm LA}$ with respect to this distribution:

[TABLE]

To demonstrate that this is a biased estimate of the true parameter $q$, evaluate this expression as a function of $q$ for different values of $a$ and the uniform prior $r=0.5$. Figure 4, showing the dependence of the expected value for $\hat{q}^{\rm LA}$ from $q$, confirms that $\hat{q}^{\rm LA}$ is a biased estimate for $q$.

Now we check whether $\hat{q}^{\rm LA}$ converges to $q$ as the number of noisy labels $n$ tends to infinity. By the law of large numbers, the fraction of ones among noisy labels converges:

[TABLE]

Assume that workers are not malicious, i.e. $a>0.5$. Note that

[TABLE]

therefore $\hat{q}^{\rm LA}$ converges to different values depending on $q$:

[TABLE]

Thus, $\hat{q}^{\rm LA}$ is not a consistent estimate for $q$ unless the distribution $\operatorname{Bernoulli}(q)$ is degenerate or $r=q=0.5$. ∎

Appendix B Data sets details

We use the following public data sets for our empirical studies:

• •

Duchenne smiles [15]. The task is to classify images into two categories – a Duchenne smile (“enjoyment” smile) and a non-Duchenne (“social” smile).

• •

Web search [18] and TREC [4]. The task is to rate query-URL pairs. For a given query-URL pair, a worker is asked to rate how relevant is the URL to the search query. For the Web search data, the rating scale has 5 levels: perfect, excellent, good, fair, or bad. For the TREC data, the rating scale is ternary: highly relevant, relevant, and non-relevant.

• •

Recognising Textual Entailment [12]. For this task each object contains two statement, and a worker judges whether one statement implies another.

• •

Temporal Ordering [12]. Each object describes two events and the task is to judge whether one event follows another.

• •

Adult content [7]. The task is to classify web pages into four categories depending on the presence of adult content on them.

• •

Price [9]. The task is to estimate prices of household items choosing one out of seven adjacent bins corresponding to different ranges of price.

Table4 provides summary statistics for all the data sets. The table includes the following columns: the size of the label set, the number of objects, the number of crowdsourcing workers, the total number of noisy labels from those workers – the size of the training set, the mean number of labels per object, the mean and the median number of labels per worker, and also the number of objects with known ground truth labels – the size of the test set.