Spectral Clustering for Crowdsourcing with Inherently Distinct Task Types

TL;DR

This paper introduces a spectral clustering method to identify task types in crowdsourcing, enabling improved label estimation by leveraging task heterogeneity, especially when worker skills vary across task types.

Contribution

It proposes a novel spectral clustering approach for multi-type tasks in crowdsourcing, demonstrating perfect recovery conditions and enhancing label inference accuracy.

Findings

Task types can be perfectly recovered with logarithmic worker-to-task ratio.

Clustering tasks by type improves label estimation performance.

The method outperforms traditional approaches in multi-type crowdsourcing scenarios.

Abstract

The Dawid-Skene model is the most widely assumed model in the analysis of crowdsourcing algorithms that estimate ground-truth labels from noisy worker responses. In this work, we are motivated by crowdsourcing applications where workers have distinct skill sets and their accuracy additionally depends on a task's type. While weighted majority vote (WMV) with a single weight vector for each worker achieves the optimal label estimation error in the Dawid-Skene model, we show that different weights for different types are necessary for a multi-type model. Focusing on the case where there are two types of tasks, we propose a spectral method to partition tasks into two groups that cluster tasks by type. Our analysis reveals that task types can be perfectly recovered if the number of workers $n$ scales logarithmically with the number of tasks $d$. Any algorithm designed for the Dawid-Skene model can then be applied independently to each type to infer the labels. Numerical experiments show how clustering tasks by type before estimating ground-truth labels enhances the performance of crowdsourcing algorithms in practical applications.