A Labeling Task Design for Supporting Algorithmic Needs: Facilitating Worker Diversity and Reducing AI Bias

TL;DR

This paper presents a task design that promotes diverse worker participation and reduces AI bias by analyzing worker behavior and feedback, leading to a more inclusive and effective labeling process for machine learning.

Contribution

It introduces a novel human-in-the-loop approach that integrates community support and machine feedback to enhance worker engagement and mitigate algorithmic bias.

Findings

Worker decision tendencies vary by background

Community support improves engagement

Machine feedback aids in bias reduction

Abstract

Studies on supervised machine learning (ML) recommend involving workers from various backgrounds in training dataset labeling to reduce algorithmic bias. Moreover, sophisticated tasks for categorizing objects in images are necessary to improve ML performance, further complicating micro-tasks. This study aims to develop a task design incorporating the fair participation of people, regardless of their specific backgrounds or task's difficulty. By collaborating with 75 labelers from diverse backgrounds for 3 months, we analyzed workers' log-data and relevant narratives to identify the task's hurdles and helpers. The findings revealed that workers' decision-making tendencies varied depending on their backgrounds. We found that the community that positively helps workers and the machine's feedback perceived by workers could make people easily engaged in works. Hence, ML's bias could be expectedly mitigated. Based on these findings, we suggest an extended human-in-the-loop approach that connects labelers, machines, and communities rather than isolating individual workers.