# Measuring the Expertise of Workers for Crowdsourcing Applications

## TL;DR

This paper introduces a new method to evaluate worker expertise in crowdsourcing by leveraging belief functions and dataset comparisons, improving accuracy over existing measures like Fagin distance.

## Contribution

It proposes an innovative expertise measure based on belief functions and demonstrates its effectiveness through real-world audio quality assessment data.

## Key findings

- The new expertise measure outperforms Fagin distance in accuracy.
- Fusing the new measure with Fagin distance enhances evaluation reliability.
- The method is validated on a real crowdsourcing dataset.

## Abstract

Crowdsourcing platforms enable companies to propose tasks to a large crowd of users. The workers receive a compensation for their work according to the serious of the tasks they managed to accomplish. The evaluation of the quality of responses obtained from the crowd remains one of the most important problems in this context. Several methods have been proposed to estimate the expertise level of crowd workers. We propose an innovative measure of expertise assuming that we possess a dataset with an objective comparison of the items concerned. Our method is based on the definition of four factors with the theory of belief functions. We compare our method to the Fagin distance on a dataset from a real experiment, where users have to assess the quality of some audio recordings. Then, we propose to fuse both the Fagin distance and our expertise measure.