Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testing

TL;DR

This study reveals significant expert disagreement in evaluating mental health AI safety, especially on critical issues like suicide, highlighting limitations of human feedback for safety assessment.

Contribution

The paper demonstrates that expert judgments in mental health AI safety are highly inconsistent, challenging the assumption that aggregated human feedback provides a reliable ground truth.

Findings

Inter-rater reliability was consistently poor among psychiatrists evaluating responses.

Disagreement was highest on safety-critical items like suicide and self-harm.

Expert disagreement reflects coherent but incompatible clinical frameworks, not measurement error.

Abstract

Learning from human feedback~(LHF) assumes that expert judgments, appropriately aggregated, yield valid ground truth for training and evaluating AI systems. We tested this assumption in mental health, where high safety stakes make expert consensus essential. Three certified psychiatrists independently evaluated LLM-generated responses using a calibrated rubric. Despite similar training and shared instructions, inter-rater reliability was consistently poor ($ICC$ $0.087$--$0.295$), falling below thresholds considered acceptable for consequential assessment. Disagreement was highest on the most safety-critical items. Suicide and self-harm responses produced greater divergence than any other category, and was systematic rather than random. One factor yielded negative reliability (Krippendorff's $\alpha = -0.203$), indicating structured disagreement worse than chance. Qualitative interviews revealed that disagreement reflects coherent but incompatible individual clinical frameworks, safety-first, engagement-centered, and culturally-informed orientations, rather than measurement error. By demonstrating that experts rely on holistic risk heuristics rather than granular factor discrimination, these findings suggest that aggregated labels function as arithmetic compromises that effectively erase grounded professional philosophies. Our results characterize expert disagreement in safety-critical AI as a sociotechnical phenomenon where professional experience introduces sophisticated layers of principled divergence. We discuss implications for reward modeling, safety classification, and evaluation benchmarks, recommending that practitioners shift from consensus-based aggregation to alignment methods that preserve and learn from expert disagreement.