GPT-4 on Clinic Depression Assessment: An LLM-Based Pilot Study

TL;DR

This pilot study investigates GPT-4's ability to classify clinical depression from patient transcripts, highlighting the importance of prompt design and parameter tuning for reliable AI-assisted mental health assessment.

Contribution

It demonstrates GPT-4's potential for depression screening and explores how prompt complexity and temperature settings influence its diagnostic accuracy.

Findings

Optimal performance at low temperature (0.0-0.2) with complex prompts

Performance variability increases with higher temperature settings

Prompt configuration critically affects model reliability in clinical tasks

Abstract

Depression has impacted millions of people worldwide and has become one of the most prevalent mental disorders. Early mental disorder detection can lead to cost savings for public health agencies and avoid the onset of other major comorbidities. Additionally, the shortage of specialized personnel is a critical issue because clinical depression diagnosis is highly dependent on expert professionals and is time consuming. In this study, we explore the use of GPT-4 for clinical depression assessment based on transcript analysis. We examine the model's ability to classify patient interviews into binary categories: depressed and not depressed. A comparative analysis is conducted considering prompt complexity (e.g., using both simple and complex prompts) as well as varied temperature settings to assess the impact of prompt complexity and randomness on the model's performance. Results indicate that GPT-4 exhibits considerable variability in accuracy and F1-Score across configurations, with optimal performance observed at lower temperature values (0.0-0.2) for complex prompts. However, beyond a certain threshold (temperature >= 0.3), the relationship between randomness and performance becomes unpredictable, diminishing the gains from prompt complexity. These findings suggest that, while GPT-4 shows promise for clinical assessment, the configuration of the prompts and model parameters requires careful calibration to ensure consistent results. This preliminary study contributes to understanding the dynamics between prompt engineering and large language models, offering insights for future development of AI-powered tools in clinical settings.