The Algorithmic Automation Problem: Prediction, Triage, and Human Effort

TL;DR

This paper broadens the understanding of automation in AI by emphasizing the importance of instance selection and error estimation, proposing a framework that improves performance in tasks like medical diagnosis.

Contribution

It introduces a general optimization framework for decision-making in automation, incorporating instance triage and error estimation to enhance AI-human collaboration.

Findings

Heuristic methods improve AI performance in medical tasks.

Effective error estimation boosts automation gains.

Framework generalizes to various AI applications.

Abstract

In a wide array of areas, algorithms are matching and surpassing the performance of human experts, leading to consideration of the roles of human judgment and algorithmic prediction in these domains. The discussion around these developments, however, has implicitly equated the specific task of prediction with the general task of automation. We argue here that automation is broader than just a comparison of human versus algorithmic performance on a task; it also involves the decision of which instances of the task to give to the algorithm in the first place. We develop a general framework that poses this latter decision as an optimization problem, and we show how basic heuristics for this optimization problem can lead to performance gains even on heavily-studied applications of AI in medicine. Our framework also serves to highlight how effective automation depends crucially on estimating both algorithmic and human error on an instance-by-instance basis, and our results show how improvements in these error estimation problems can yield significant gains for automation as well.