On Optimizing Human-Machine Task Assignments

TL;DR

This paper investigates methods to enhance the accuracy and reduce the cost of crowdsourcing when integrated with off-the-shelf machine classifiers, by reordering tasks and jointly optimizing system parameters.

Contribution

It introduces two strategies—task reordering and joint optimization—to improve performance without deep integration of machine classifiers.

Findings

Reordering tasks significantly improves accuracy.

Joint optimization outperforms greedy parameter selection.

The proposed methods enhance system performance in real-world settings.

Abstract

When crowdsourcing systems are used in combination with machine inference systems in the real world, they benefit the most when the machine system is deeply integrated with the crowd workers. However, if researchers wish to integrate the crowd with "off-the-shelf" machine classifiers, this deep integration is not always possible. This work explores two strategies to increase accuracy and decrease cost under this setting. First, we show that reordering tasks presented to the human can create a significant accuracy improvement. Further, we show that greedily choosing parameters to maximize machine accuracy is sub-optimal, and joint optimization of the combined system improves performance.