Reasonable uncertainty: Confidence intervals in empirical Bayes discrimination detection

TL;DR

This paper examines uncertainty in empirical Bayes discrimination detection, emphasizing the importance of accounting for sampling variability and proposing a counterfactual odds-ratio to better interpret real-world discrimination.

Contribution

It introduces a new counterfactual odds-ratio estimand and highlights the significance of incorporating sampling uncertainty in empirical Bayes discrimination detection.

Findings

Sampling uncertainty affects empirical findings.

Counterfactual odds-ratio offers interpretable discrimination measure.

Careful uncertainty quantification is crucial for reliable analysis.

Abstract

We revisit empirical Bayes discrimination detection, focusing on uncertainty arising from both partial identification and sampling variability. While prior work has mostly focused on partial identification, we find that some empirical findings are not robust to sampling uncertainty. To better connect statistical evidence to the magnitude of real-world discriminatory behavior, we propose a counterfactual odds-ratio estimand with a attractive properties and interpretation. Our analysis reveals the importance of careful attention to uncertainty quantification and downstream goals in empirical Bayes analyses.