Fairness Audits of Institutional Risk Models in Deployed ML Pipelines

TL;DR

This paper presents a methodology for auditing fairness in institutional risk models, revealing disparities across demographic groups and how they are amplified through the ML pipeline.

Contribution

It introduces a replicable audit approach for institutional ML systems, highlighting how disparities emerge and are compounded across stages.

Findings

Younger, male, and international students are disproportionately flagged for support.

Post-processing amplifies disparities by collapsing probabilities into risk tiers.

Disparities exist across training data, model predictions, and post-processing stages.

Abstract

Fairness audits of institutional risk models are critical for understanding how deployed machine learning pipelines allocate resources. Drawing on multi-year collaboration with Centennial College, where our prior ethnographic work introduced the ASP-HEI Cycle, we present a replica-based audit of a deployed Early Warning System (EWS), replicating its model using institutional training data and design specifications. We evaluate disparities by gender, age, and residency status across the full pipeline (training data, model predictions, and post-processing) using standard fairness metrics. Our audit reveals systematic misallocation: younger, male, and international students are disproportionately flagged for support, even when many ultimately succeed, while older and female students with comparable dropout risk are under-identified. Post-processing amplifies these disparities by collapsing heterogeneous probabilities into percentile-based risk tiers. This work provides a replicable methodology for auditing institutional ML systems and shows how disparities emerge and compound across stages, highlighting the importance of evaluating construct validity alongside statistical fairness. It contributes one empirical thread to a broader program investigating algorithms, student data, and power in higher education.