Operating Imperfect AI: Reliability Drift and Human Congestion

TL;DR

This paper models the management of human-in-the-loop AI systems facing reliability drift and human congestion as a dynamic control problem, deriving optimal policies and identifying critical thresholds for system safety and performance.

Contribution

It introduces a novel queueing control framework for imperfect AI-human systems, establishing structural properties and a capacity phase transition analysis.

Findings

Optimal escalation policy driven by shadow price of capacity.

Monotonicity results: congestion shedding and safety buffering.

Identification of a capacity phase transition point.

Abstract

The deployment of machine learning in high-stakes services relies on ``human-in-the-loop'' architectures to mitigate algorithmic uncertainty. However, existing static policies fail to address a fundamental tension: algorithms suffer from stochastic ``reliability drift,'' while human override capacity is scarce and congestible. We formulate the management of such systems as a dynamic queueing control problem. The system state is defined by the tuple (queue backlog, reliability regime), and the control variable is a state-dependent risk threshold. We prove that the optimal escalation policy is driven by the endogenous ``Shadow Price of Capacity.'' We establish two key structural monotonicity results: (i) Congestion Shedding, where the threshold rises with backlog to sacrifice marginal accuracy for responsiveness; and (ii) Safety Buffering, where the threshold lowers during drift to use the queue as a ``risk capacitor.'' Furthermore, we identify a critical ``Capacity Phase Transition'' in the arrival-drift parameter space, beyond which no policy can maintain safety standards without causing structural system failure (infinite queues). Our results provide rigorous operational rules for managing the interface between imperfect algorithms and congested experts.