All Required, In Order: Phase-Level Evaluation for AI-Human Dialogue in Healthcare and Beyond

TL;DR

This paper introduces OIP-SCE, a novel evaluation method for conversational AI in healthcare that assesses compliance with clinical obligations in the correct order, providing transparent, actionable, and auditable results.

Contribution

The paper presents OIP-SCE, a new phase-level evaluation framework that ensures compliance with clinical obligations, bridging the gap between AI capabilities and healthcare needs.

Findings

OIP-SCE effectively evaluates clinical compliance in dialogue.

The method improves transparency and auditability of AI in healthcare.

Case studies demonstrate practical application and benefits.

Abstract

Conversational AI is starting to support real clinical work, but most evaluation methods miss how compliance depends on the full course of a conversation. We introduce Obligatory-Information Phase Structured Compliance Evaluation (OIP-SCE), an evaluation method that checks whether every required clinical obligation is met, in the right order, with clear evidence for clinicians to review. This makes complex rules practical and auditable, helping close the gap between technical progress and what healthcare actually needs. We demonstrate the method in two case studies (respiratory history, benefits verification) and show how phase-level evidence turns policy into shared, actionable steps. By giving clinicians control over what to check and engineers a clear specification to implement, OIP-SCE provides a single, auditable evaluation surface that aligns AI capability with clinical workflow and supports routine, safe use.