The Doctor Will (Still) See You Now: On the Structural Limits of Agentic AI in Healthcare

TL;DR

This paper examines the limitations of autonomous agentic AI in healthcare, highlighting tensions between promises and practical constraints, and emphasizing safety, accountability, and sociotechnical factors.

Contribution

It provides a qualitative analysis of stakeholder perspectives, revealing conceptual, operational, and evaluative tensions in deploying agentic AI in healthcare.

Findings

Identifies conceptual fragmentation of 'agentic' AI

Highlights the autonomy contradiction between promises and reality

Discovers evaluation blind spots prioritizing technical metrics

Abstract

Across healthcare, agentic artificial intelligence (AI) systems are increasingly promoted as capable of autonomous action, yet in practice they currently operate under near-total human oversight due to safety, regulatory, and liability constraints that make autonomous clinical reasoning infeasible in high-stakes environments. While market enthusiasm suggests a revolution in healthcare agents, the conceptual assumptions and accountability structures shaping these systems remain underexamined. We present a qualitative study based on interviews with 20 stakeholders, including developers, implementers, and end users. Our analysis identifies three mutually reinforcing tensions: conceptual fragmentation regarding the definition of `agentic'; an autonomy contradiction where commercial promises exceed operational reality; and an evaluation blind spot that prioritizes technical benchmarks over sociotechnical safety. We argue that agentic {AI} functions as a site of contested meaning-making where technical aspirations, commercial incentives, and clinical constraints intersect, carrying material consequences for patient safety and the distribution of blame.