Open-Set Crowdsourcing using Multiple-Source Transfer Learning

TL;DR

This paper introduces OSCrowd, a novel method for open set crowdsourcing that leverages multiple-source transfer learning to infer labels and guide annotations in unfamiliar tasks with unknown label spaces.

Contribution

OSCrowd is a new approach that integrates multiple source datasets and transfer learning to handle open set crowdsourcing scenarios with unknown label spaces.

Findings

OSCrowd effectively infers label spaces in open set crowdsourcing.

It outperforms existing crowdsourcing methods in experimental validation.

The approach successfully guides crowd workers in unfamiliar task domains.

Abstract

We raise and define a new crowdsourcing scenario, open set crowdsourcing, where we only know the general theme of an unfamiliar crowdsourcing project, and we don't know its label space, that is, the set of possible labels. This is still a task annotating problem, but the unfamiliarity with the tasks and the label space hampers the modelling of the task and of workers, and also the truth inference. We propose an intuitive solution, OSCrowd. First, OSCrowd integrates crowd theme related datasets into a large source domain to facilitate partial transfer learning to approximate the label space inference of these tasks. Next, it assigns weights to each source domain based on category correlation. After this, it uses multiple-source open set transfer learning to model crowd tasks and assign possible annotations. The label space and annotations given by transfer learning will be used to guide and standardize crowd workers' annotations. We validate OSCrowd in an online scenario, and prove that OSCrowd solves the open set crowdsourcing problem, works better than related crowdsourcing solutions.