Multi-Label Classification with Generative AI Models in Healthcare: A Case Study of Suicidality and Risk Factors

TL;DR

This study demonstrates the effectiveness of generative large language models like GPT-3.5 and GPT-4.5 in multi-label classification of suicidality-related factors from electronic health records, advancing clinical AI applications.

Contribution

It introduces a novel end-to-end generative multi-label classification pipeline and advanced evaluation methods for clinical text analysis using LLMs.

Findings

GPT-3.5 achieved 0.94 partial match accuracy and 0.91 F1 score.

GPT-4.5 with guided prompting outperformed GPT-3.5, especially on rare labels.

Models tend to over-label and conflate related suicidality factors.

Abstract

Suicide remains a pressing global health crisis, with over 720,000 deaths annually and millions more affected by suicide ideation (SI) and suicide attempts (SA). Early identification of suicidality-related factors (SrFs), including SI, SA, exposure to suicide (ES), and non-suicidal self-injury (NSSI), is critical for timely intervention. While prior studies have applied AI to detect SrFs in clinical notes, most treat suicidality as a binary classification task, overlooking the complexity of cooccurring risk factors. This study explores the use of generative large language models (LLMs), specifically GPT-3.5 and GPT-4.5, for multi-label classification (MLC) of SrFs from psychiatric electronic health records (EHRs). We present a novel end to end generative MLC pipeline and introduce advanced evaluation methods, including label set level metrics and a multilabel confusion matrix for error analysis. Finetuned GPT-3.5 achieved top performance with 0.94 partial match accuracy and 0.91 F1 score, while GPT-4.5 with guided prompting showed superior performance across label sets, including rare or minority label sets, indicating a more balanced and robust performance. Our findings reveal systematic error patterns, such as the conflation of SI and SA, and highlight the models tendency toward cautious over labeling. This work not only demonstrates the feasibility of using generative AI for complex clinical classification tasks but also provides a blueprint for structuring unstructured EHR data to support large scale clinical research and evidence based medicine.