Auditing a Dutch Public Sector Risk Profiling Algorithm Using an Unsupervised Bias Detection Tool

TL;DR

This paper demonstrates how an unsupervised bias detection tool can identify biases in a large-scale Dutch government algorithm used for student risk scoring, even without demographic data, and provides an open-source tool for bias auditing.

Contribution

It introduces an unsupervised bias detection method applied to real-world, high-stakes government decision-making and releases an open-source library for bias audits.

Findings

Detected known disparities between demographic groups

Audited over 250,000 students nationwide

Validated the effectiveness of the bias detection tool

Abstract

Algorithms are increasingly used to automate or aid human decisions, yet recent research shows that these algorithms may exhibit bias across legally protected demographic groups. However, data on these groups may be unavailable to organizations or external auditors due to privacy legislation. This paper studies bias detection using an unsupervised bias detection tool when data on demographic groups are unavailable. We collaborated with the Dutch Executive Agency for Education to audit an algorithm that was used to assign risk scores to college students at the national level in the Netherlands between 2012-2023. Our audit covers more than 250,000 students across the country. The unsupervised bias detection tool highlights known disparities between students with a non-European migration background and students with a Dutch or European-migration background. Our contributions are two-fold: (1) we assess bias in a real-world, large-scale, and high-stakes decision-making process by a governmental organization; (2) we provide the unsupervised bias detection tool in an open-source library for others to use to complete bias audits. Our work serves as a starting point for a deliberative assessment by human experts to evaluate potential discrimination in algorithmic decision-making.