Toward a Perspectivist Turn in Ground Truthing for Predictive Computing

TL;DR

This paper proposes a shift from traditional gold standard datasets to a perspectivist approach in supervised machine learning, emphasizing the integration of human perspectives to improve data annotation and model evaluation.

Contribution

It introduces the concept of data perspectivism, advocating for incorporating human opinions in ML data annotation, and discusses its potential benefits and implementation strategies.

Findings

Perspectivism can enhance subjective and objective ML tasks.

Advantages include more nuanced data representation and reduced bias.

Potential challenges involve complexity in data integration.

Abstract

Most Artificial Intelligence applications are based on supervised machine learning (ML), which ultimately grounds on manually annotated data. The annotation process is often performed in terms of a majority vote and this has been proved to be often problematic, as highlighted by recent studies on the evaluation of ML models. In this article we describe and advocate for a different paradigm, which we call data perspectivism, which moves away from traditional gold standard datasets, towards the adoption of methods that integrate the opinions and perspectives of the human subjects involved in the knowledge representation step of ML processes. Drawing on previous works which inspired our proposal we describe the potential of our proposal for not only the more subjective tasks (e.g. those related to human language) but also to tasks commonly understood as objective (e.g. medical decision making), and present the main advantages of adopting a perspectivist stance in ML, as well as possible disadvantages, and various ways in which such a stance can be implemented in practice. Finally, we share a set of recommendations and outline a research agenda to advance the perspectivist stance in ML.