CHiRPE: A Step Towards Real-World Clinical NLP with Clinician-Oriented Model Explanations

TL;DR

CHiRPE is an NLP pipeline designed for clinical use that predicts psychosis risk from interviews and provides clinician-aligned explanations, improving interpretability and accuracy in mental health assessments.

Contribution

This work introduces CHiRPE, a novel clinical NLP pipeline that combines symptom mapping, LLM summarization, and BERT classification with clinician-developed explanations.

Findings

Achieved over 90% accuracy with multiple BERT models

Clinicians preferred the novel explanation formats

Outperformed baseline models in psychosis risk prediction

Abstract

The medical adoption of NLP tools requires interpretability by end users, yet traditional explainable AI (XAI) methods are misaligned with clinical reasoning and lack clinician input. We introduce CHiRPE (Clinical High-Risk Prediction with Explainability), an NLP pipeline that takes transcribed semi-structured clinical interviews to: (i) predict psychosis risk; and (ii) generate novel SHAP explanation formats co-developed with clinicians. Trained on 944 semi-structured interview transcripts across 24 international clinics of the AMP-SCZ study, the CHiRPE pipeline integrates symptom-domain mapping, LLM summarisation, and BERT classification. CHiRPE achieved over 90% accuracy across three BERT variants and outperformed baseline models. Explanation formats were evaluated by 28 clinical experts who indicated a strong preference for our novel concept-guided explanations, especially hybrid graph-and-text summary formats. CHiRPE demonstrates that clinically-guided model development produces both accurate and interpretable results. Our next step is focused on real-world testing across our 24 international sites.