Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studies

TL;DR

This paper explores the history of social science audit studies to inform the development of algorithmic audits that are socially just, emphasizing lessons from past practices to address current challenges in assessing sociotechnical systems.

Contribution

It analyzes the historical evolution of audit studies in social sciences to identify lessons and tensions relevant for creating socially just algorithmic auditing practices.

Findings

Historical audit studies emphasized social justice and community involvement.

Modern algorithmic audits can benefit from participatory and justice-oriented approaches.

Lessons from social science audits highlight tensions between objectivity and social engagement.

Abstract

Algorithmic audits have been embraced as tools to investigate the functioning and consequences of sociotechnical systems. Though the term is used somewhat loosely in the algorithmic context and encompasses a variety of methods, it maintains a close connection to audit studies in the social sciences--which have, for decades, used experimental methods to measure the prevalence of discrimination across domains like housing and employment. In the social sciences, audit studies originated in a strong tradition of social justice and participatory action, often involving collaboration between researchers and communities; but scholars have argued that, over time, social science audits have become somewhat distanced from these original goals and priorities. We draw from this history in order to highlight difficult tensions that have shaped the development of social science audits, and to assess their implications in the context of algorithmic auditing. In doing so, we put forth considerations to assist in the development of robust and engaged assessments of sociotechnical systems that draw from auditing's roots in racial equity and social justice.