ChatGPT for Suicide Risk Assessment on Social Media: Quantitative Evaluation of Model Performance, Potentials and Limitations

TL;DR

This study evaluates ChatGPT's effectiveness in assessing suicidality from social media posts, comparing it with fine-tuned models, and explores how hyperparameter tuning influences its performance and potential clinical utility.

Contribution

It introduces a framework for quantitative evaluation of ChatGPT in suicidality assessment and analyzes the effects of temperature settings on its responses.

Findings

ChatGPT achieves notable accuracy but is outperformed by fine-tuned models.

Adjusting temperature parameters can enhance ChatGPT's response quality.

The framework facilitates systematic comparison of large language models in mental health tasks.

Abstract

This paper presents a novel framework for quantitatively evaluating the interactive ChatGPT model in the context of suicidality assessment from social media posts, utilizing the University of Maryland Reddit suicidality dataset. We conduct a technical evaluation of ChatGPT's performance on this task using Zero-Shot and Few-Shot experiments and compare its results with those of two fine-tuned transformer-based models. Additionally, we investigate the impact of different temperature parameters on ChatGPT's response generation and discuss the optimal temperature based on the inconclusiveness rate of ChatGPT. Our results indicate that while ChatGPT attains considerable accuracy in this task, transformer-based models fine-tuned on human-annotated datasets exhibit superior performance. Moreover, our analysis sheds light on how adjusting the ChatGPT's hyperparameters can improve its ability to assist mental health professionals in this critical task.