A quality of mercy is not trained: the imagined vs. the practiced in healthcare process-specialized AI development

TL;DR

This paper examines how the design choices in healthcare AI systems, especially in surgery scheduling, influence ethical considerations by comparing imagined and actual practices, highlighting the need for more context-aware development.

Contribution

It reveals how early representational decisions in AI design can limit ethical considerations, advocating for a more situated approach in healthcare AI development.

Findings

Early design decisions exclude key ethical dimensions.

AI systems tend to reflect simplified, rule-based models.

Practiced coordination involves ethical discretion often absent in AI.

Abstract

In high stakes organizational contexts like healthcare, artificial intelligence (AI) systems are increasingly being designed to augment complex coordination tasks. This paper investigates how the ethical stakes of such systems are shaped by their epistemic framings: what aspects of work they represent, and what they exclude. Drawing on an embedded study of AI development for operating room (OR) scheduling at a Canadian hospital, we compare scheduling-as-imagined in the AI design process: rule-bound, predictable, and surgeon-centric, with scheduling-as-practiced as a fluid, patient-facing coordination process involving ethical discretion. We show how early representational decisions narrowed what the AI could support, resulting in epistemic foreclosure: the premature exclusion of key ethical dimensions from system design. Our findings surface the moral consequences of abstraction and call for a more situated approach to designing healthcare process-specialized artificial intelligence systems.