CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?

TL;DR

This paper introduces CHI-Bench, a comprehensive healthcare workflow benchmark testing AI agents on complex, multi-role, long-horizon tasks involving policy-rich decision-making and multi-turn interactions.

Contribution

It presents a new benchmark with high-fidelity simulations and detailed tasks to evaluate AI performance on realistic healthcare workflows, highlighting current limitations.

Findings

Best agent resolves only 28% of tasks

No agent clears 20% on strict pass criteria

Performance drops to 3.8% when executing all tasks in a single session

Abstract

End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks: policy density, decisions must be grounded in a large library of medical, insurance, and operational rules; Multi-role composition: a single task requires the agent to play multiple roles with handoffs; and multilateral interaction: intermediate workflow steps are multi-turn dialogs, such as peer-to-peer review and patient outreach. We introduce $\chi$-Bench, a benchmark of long-horizon healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, which it must drive to a terminal status through tool calls and writing the role's artifacts, guided by a 1,290+ document managed-care operations handbook skill. Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass^3, and executing all tasks in a single session slumps the performance to 3.8%. These results raise the hypothesis that similar gaps are likely to surface in other policy-dense, role-composed, irreversible enterprise domains.