Statistical Tests for Replacing Human Decision Makers with Algorithms

TL;DR

This paper introduces a statistical framework for replacing human decision makers with AI algorithms, demonstrating improved accuracy in medical diagnoses through empirical testing on nationwide data.

Contribution

It develops a novel statistical approach combining frequentist and Bayesian methods to benchmark and replace human decisions with machine predictions.

Findings

Higher true positive rate with AI algorithms

Lower false positive rate compared to doctors

Effective in abnormal birth detection

Abstract

This paper proposes a statistical framework of using artificial intelligence to improve human decision making. The performance of each human decision maker is benchmarked against that of machine predictions. We replace the diagnoses made by a subset of the decision makers with the recommendation from the machine learning algorithm. We apply both a heuristic frequentist approach and a Bayesian posterior loss function approach to abnormal birth detection using a nationwide dataset of doctor diagnoses from prepregnancy checkups of reproductive age couples and pregnancy outcomes. We find that our algorithm on a test dataset results in a higher overall true positive rate and a lower false positive rate than the diagnoses made by doctors only.