Wisdom for the Crowd: Discoursive Power in Annotation Instructions for Computer Vision

TL;DR

This paper examines how annotation instructions in computer vision outsourcing reflect and reinforce social inequalities and power dynamics, impacting dataset quality and societal biases.

Contribution

It reveals how task instructions encode worldviews and for-profit motives, shaping labor practices and perpetuating social inequalities in datasets.

Findings

Annotations reflect imposed worldviews and social biases.

For-profit goals influence annotation instructions.

Power asymmetries are embedded in annotation practices.

Abstract

Developers of computer vision algorithms outsource some of the labor involved in annotating training data through business process outsourcing companies and crowdsourcing platforms. Many data annotators are situated in the Global South and are considered independent contractors. This paper focuses on the experiences of Argentinian and Venezuelan annotation workers. Through qualitative methods, we explore the discourses encoded in the task instructions that these workers follow to annotate computer vision datasets. Our preliminary findings indicate that annotation instructions reflect worldviews imposed on workers and, through their labor, on datasets. Moreover, we observe that for-profit goals drive task instructions and that managers and algorithms make sure annotations are done according to requesters' commands. This configuration presents a form of commodified labor that perpetuates power asymmetries while reinforcing social inequalities and is compelled to reproduce them into datasets and, subsequently, in computer vision systems.