Measuring Epistemic Unfairness for Algorithmic Decision-Making

TL;DR

This paper introduces a quantitative framework to measure epistemic injustices in algorithmic decision-making, addressing a gap in fairness assessments beyond traditional error-based metrics.

Contribution

It proposes a deficit-based model and adapts fairness indices to evaluate epistemic harms, enabling more comprehensive auditing of algorithmic systems.

Findings

The framework identifies signatures of unfairness like exclusionary tails and hierarchical concentration.

It supports longitudinal auditing through simulation of opinion dynamics.

The approach makes epistemic harms explicit for system design and evaluation.

Abstract

Algorithmic systems increasingly function as epistemic infrastructures that govern the conditions of interpretative access and social belief. Yet, mainstream auditing strategies operationalize fairness primarily in predictive terms - error rates, calibration, or group-level parity - leaving epistemic harms under-theorized and under-measured. We propose a quantitative framework for evaluating forms of epistemic injustice in algorithmic environments. First, we introduce a deficit-based template that models epistemic injustices as gaps between ideal and realized conditions across features such as credibility, uptake, and epistemic agency. We map these deficits to concrete stages of algorithmic mediation, showing how epistemic injustice can persist even when standard fairness constraints are satisfied. Drawing on distributive fairness indices, we distinguish two evaluation stances: resource inequality, where indices are applied to distributions of epistemic goods directly, and capability/rights inequity, where indices are applied to output-induced epistemic opportunity. We provide an epistemic translation of canonical indices, illustrating how they diagnose complementary signatures of unfairness - such as exclusionary tails and hierarchical concentration - and support longitudinal auditing under iterative deployment. We also provide a simulation study of a recommender-mediated opinion dynamics setting, showing how the proposed indices capture the evolution of epistemic unfairness under repeated platform interventions. The result is a measurement framework that makes the epistemic dimension of algorithmic harms explicit for system design and evaluation.