Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision

TL;DR

This paper explores how power relations and societal influences shape data annotation practices in computer vision, revealing that annotations are often influenced by impositions and values of higher actors, impacting societal outcomes.

Contribution

It provides a constructivist grounded theory analysis of annotation practices, highlighting the influence of power dynamics and naturalized impositions in data labeling processes.

Findings

Annotators' work is shaped by interests and values of higher actors.

Arbitrary classifications are imposed and naturalized.

Data annotation is an exercise of power with societal implications.

Abstract

The interpretation of data is fundamental to machine learning. This paper investigates practices of image data annotation as performed in industrial contexts. We define data annotation as a sense-making practice, where annotators assign meaning to data through the use of labels. Previous human-centered investigations have largely focused on annotators subjectivity as a major cause for biased labels. We propose a wider view on this issue: guided by constructivist grounded theory, we conducted several weeks of fieldwork at two annotation companies. We analyzed which structures, power relations, and naturalized impositions shape the interpretation of data. Our results show that the work of annotators is profoundly informed by the interests, values, and priorities of other actors above their station. Arbitrary classifications are vertically imposed on annotators, and through them, on data. This imposition is largely naturalized. Assigning meaning to data is often presented as a technical matter. This paper shows it is, in fact, an exercise of power with multiple implications for individuals and society.