WEIRD Audits? Research Trends, Linguistic and Geographical Disparities in the Algorithm Audits of Online Platforms -- A Systematic Literature Review

TL;DR

This systematic review analyzes 176 studies on algorithm audits of online platforms, highlighting geographic, linguistic, and methodological disparities, and emphasizing the need for more inclusive and diverse research practices.

Contribution

The paper provides a comprehensive overview of current trends and gaps in online platform algorithm auditing research, emphasizing disparities and areas for improvement.

Findings

Research heavily focuses on US and English-language data.

Most studies examine a limited set of group-based attributes.

There is a tendency to simplify complex biases in audits.

Abstract

The increasing reliance on complex algorithmic systems by online platforms has sparked a growing need for algorithm auditing, a methodology evaluating these systems' functionality and impact. In this paper, we systematically review 176 peer-reviewed online platform-focused algorithm auditing studies and identify trends in their methodological approaches, the geographic distribution of authors, and the selection of platforms, languages, geographies, and group-based attributes in the focus of the reviewed research. We find a significant skew of research focus towards few online platforms, Western contexts, particularly the US, and English language data. Additionally, our analysis indicates a tendency to focus on a narrow set of group-based attributes, often operationalized in simplified ways, which might obscure more nuanced aspects of algorithmic bias and discrimination. We provide a clearer understanding of the current state of the online platform-focused algorithm auditing and identify gaps to be addressed for a more inclusive and representative research landscape.