Bias Delayed is Bias Denied? Assessing the Effect of Reporting Delays on Disparity Assessments

TL;DR

This paper investigates how delays in reporting demographic data affect the accuracy of disparity assessments in healthcare, revealing significant impacts and limited mitigation options, with implications for fairness evaluations across domains.

Contribution

It characterizes the effects of reporting delays on disparity assessments using real healthcare data and evaluates the limitations of imputation methods in correcting these delays.

Findings

Reporting delays vary widely across demographic groups.

Delays significantly distort disparity assessments in healthcare outcomes.

Imputation methods have limited effectiveness in correcting for reporting delays.

Abstract

Conducting disparity assessments at regular time intervals is critical for surfacing potential biases in decision-making and improving outcomes across demographic groups. Because disparity assessments fundamentally depend on the availability of demographic information, their efficacy is limited by the availability and consistency of available demographic identifiers. While prior work has considered the impact of missing data on fairness, little attention has been paid to the role of delayed demographic data. Delayed data, while eventually observed, might be missing at the critical point of monitoring and action -- and delays may be unequally distributed across groups in ways that distort disparity assessments. We characterize such impacts in healthcare, using electronic health records of over 5M patients across primary care practices in all 50 states. Our contributions are threefold. First, we document the high rate of race and ethnicity reporting delays in a healthcare setting and demonstrate widespread variation in rates at which demographics are reported across different groups. Second, through a set of retrospective analyses using real data, we find that such delays impact disparity assessments and hence conclusions made across a range of consequential healthcare outcomes, particularly at more granular levels of state-level and practice-level assessments. Third, we find limited ability of conventional methods that impute missing race in mitigating the effects of reporting delays on the accuracy of timely disparity assessments. Our insights and methods generalize to many domains of algorithmic fairness where delays in the availability of sensitive information may confound audits, thus deserving closer attention within a pipeline-aware machine learning framework.