The Capability of Large Language Models to Measure Psychiatric Functioning

TL;DR

This study evaluates Med-PaLM 2, a large language model trained on medical data, for its ability to predict psychiatric functioning from clinical descriptions, showing promising accuracy comparable to human raters.

Contribution

It demonstrates that Med-PaLM 2 can assess psychiatric functioning across various disorders without specific training for this task, highlighting its potential in clinical settings.

Findings

Med-PaLM 2 accurately predicts depression scores with 0.80-0.84 accuracy.

Performance on depression scores is statistically similar to human clinicians.

The model shows potential for flexible psychiatric risk assessment from free-text descriptions.

Abstract

The current work investigates the capability of Large language models (LLMs) that are explicitly trained on large corpuses of medical knowledge (Med-PaLM 2) to predict psychiatric functioning from patient interviews and clinical descriptions without being trained to do so. To assess this, n = 145 depression and n =115 PTSD assessments and n = 46 clinical case studies across high prevalence/high comorbidity disorders (Depressive, Anxiety, Psychotic, trauma and stress, Addictive disorders) were analyzed using prompts to extract estimated clinical scores and diagnoses. Results demonstrate that Med-PaLM 2 is capable of assessing psychiatric functioning across a range of psychiatric conditions with the strongest performance being the prediction of depression scores based on standardized assessments (Accuracy range= 0.80 - 0.84) which were statistically indistinguishable from human clinical raters t(1,144) = 1.20; p = 0.23. Results show the potential for general clinical language models to flexibly predict psychiatric risk based on free descriptions of functioning from both patients and clinicians.