Mixture of Experts based Multi-task Supervise Learning from Crowds

TL;DR

This paper introduces a novel multi-task supervised learning framework for crowdsourcing truth inference that models worker behavior at the item feature level, improving accuracy over existing methods.

Contribution

It proposes a new paradigm that avoids modeling ground truth directly and introduces a Mixture of Experts model for worker behavior at the item feature level.

Findings

MMLC-owf outperforms state-of-the-art truth inference methods.

MMLC-df improves existing truth inference accuracy.

Experimental results validate the effectiveness of the proposed models.

Abstract

Existing truth inference methods in crowdsourcing aim to map redundant labels and items to the ground truth. They treat the ground truth as hidden variables and use statistical or deep learning-based worker behavior models to infer the ground truth. However, worker behavior models that rely on ground truth hidden variables overlook workers' behavior at the item feature level, leading to imprecise characterizations and negatively impacting the quality of truth inference. This paper proposes a new paradigm of multi-task supervised learning from crowds, which eliminates the need for modeling of items's ground truth in worker behavior models. Within this paradigm, we propose a worker behavior model at the item feature level called Mixture of Experts based Multi-task Supervised Learning from Crowds (MMLC). Two truth inference strategies are proposed within MMLC. The first strategy, named MMLC-owf, utilizes clustering methods in the worker spectral space to identify the projection vector of the oracle worker. Subsequently, the labels generated based on this vector are considered as the inferred truth. The second strategy, called MMLC-df, employs the MMLC model to fill the crowdsourced data, which can enhance the effectiveness of existing truth inference methods. Experimental results demonstrate that MMLC-owf outperforms state-of-the-art methods and MMLC-df enhances the quality of existing truth inference methods.