Falsifying Predictive Algorithm

TL;DR

This paper introduces a statistical falsification framework to verify if predictive algorithms are estimating intended outcomes, helping identify unintended biases before deployment using minimal assumptions.

Contribution

It develops a novel, principled testing method for discriminant validity applicable with multiple proxies and minimal assumptions, inspired by causal inference and econometrics.

Findings

Framework successfully detects discriminant validity in admissions data

Identifies lack of discriminant validity with respect to race in the admissions setting

Highlights limitations of the approach in criminal justice applications

Abstract

Empirical investigations into unintended model behavior often show that the algorithm is predicting another outcome than what was intended. These exposes highlight the need to identify when algorithms predict unintended quantities - ideally before deploying them into consequential settings. We propose a falsification framework that provides a principled statistical test for discriminant validity: the requirement that an algorithm predict intended outcomes better than impermissible ones. Drawing on falsification practices from causal inference, econometrics, and psychometrics, our framework compares calibrated prediction losses across outcomes to assess whether the algorithm exhibits discriminant validity with respect to a specified impermissible proxy. In settings where the target outcome is difficult to observe, multiple permissible proxy outcomes may be available; our framework accommodates both this setting and the case with a single permissible proxy. Throughout we use nonparametric hypothesis testing methods that make minimal assumptions on the data-generating process. We illustrate the method in an admissions setting, where the framework establishes discriminant validity with respect to gender but fails to establish discriminant validity with respect to race. This demonstrates how falsification can serve as an early validity check, prior to fairness or robustness analyses. We also provide analysis in a criminal justice setting, where we highlight the limitations of our framework and emphasize the need for complementary approaches to assess other aspects of construct validity and external validity.