Evaluating Complex Task through Crowdsourcing: Multiple Views Approach

TL;DR

This paper introduces a multi-view crowdsourcing framework for grading complex tasks, addressing knowledge limitations of crowd workers by using expert-defined perspectives, bias detection, and aggregation to improve grading accuracy.

Contribution

It proposes a novel multi-view grading framework with bias detection and correction, enhancing accuracy for complex tasks in crowdsourcing environments.

Findings

The model outperforms traditional approaches without multiple views.

Bias detection and debiasing improve grading accuracy.

Synthetic data experiments validate the effectiveness of the approach.

Abstract

With the popularity of massive open online courses, grading through crowdsourcing has become a prevalent approach towards large scale classes. However, for getting grades for complex tasks, which require specific skills and efforts for grading, crowdsourcing encounters a restriction of insufficient knowledge of the workers from the crowd. Due to knowledge limitation of the crowd graders, grading based on partial perspectives becomes a big challenge for evaluating complex tasks through crowdsourcing. Especially for those tasks which not only need specific knowledge for grading, but also should be graded as a whole instead of being decomposed into smaller and simpler subtasks. We propose a framework for grading complex tasks via multiple views, which are different grading perspectives defined by experts for the task, to provide uniformity. Aggregation algorithm based on graders variances are used to combine the grades for each view. We also detect bias patterns of the graders, and debias them regarding each view of the task. Bias pattern determines how the behavior is biased among graders, which is detected by a statistical technique. The proposed approach is analyzed on a synthetic data set. We show that our model gives more accurate results compared to the grading approaches without different views and debiasing algorithm.