Modeling Expert AI Diagnostic Alignment via Immutable Inference Snapshots

TL;DR

This paper presents a structured framework for analyzing human-in-the-loop validation in clinical AI, using immutable inference snapshots and a multi-level concordance evaluation to better understand diagnostic alignment and correction dynamics.

Contribution

It introduces a diagnostic alignment framework with immutable inference states and a comprehensive concordance evaluation, advancing structured analysis of expert validation in clinical AI.

Findings

Exact agreement rate was 71.4% and stable under semantic similarity.

Structured cross-category analysis achieved 100% comprehensive concordance.

Binary lexical evaluation underestimates clinically meaningful alignment.

Abstract

Human-in-the-loop validation is essential in safety-critical clinical AI, yet the transition between initial model inference and expert correction is rarely analyzed as a structured signal. We introduce a diagnostic alignment framework in which the AI-generated image based report is preserved as an immutable inference state and systematically compared with the physician-validated outcome. The inference pipeline integrates a vision-enabled large language model, BERT- based medical entity extraction, and a Sequential Language Model Inference (SLMI) step to enforce domain-consistent refinement prior to expert review. Evaluation on 21 dermatological cases (21 complete AI physician pairs) em- ployed a four-level concordance framework comprising exact primary match rate (PMR), semantic similarity-adjusted rate (AMR), cross-category alignment, and Comprehensive Concordance Rate (CCR). Exact agreement reached 71.4% and remained unchanged under semantic similarity (t = 0.60), while structured cross-category and differential overlap analysis yielded 100% comprehensive concordance (95% CI: [83.9%, 100%]). No cases demonstrated complete diagnostic divergence. These findings show that binary lexical evaluation substantially un- derestimates clinically meaningful alignment. Modeling expert validation as a structured transformation enables signal-aware quantification of correction dynamics and supports traceable, human aligned evaluation of image based clinical decision support systems.