Evaluating Transformer Models for Suicide Risk Detection on Social Media

TL;DR

This study evaluates transformer models for detecting suicide risk in social media posts, showing that minimal tuning of general-purpose models like GPT-4o can achieve high accuracy and competitive results.

Contribution

The paper demonstrates that fine-tuned GPT-4o outperforms other models in suicide risk detection, highlighting the effectiveness of minimal tuning of large language models for this task.

Findings

GPT-4o achieved second place in the competition.

Fine-tuned GPT-4o outperformed other configurations.

Minimal tuning of general models can yield state-of-the-art results.

Abstract

The detection of suicide risk in social media is a critical task with potential life-saving implications. This paper presents a study on leveraging state-of-the-art natural language processing solutions for identifying suicide risk in social media posts as a submission for the "IEEE BigData 2024 Cup: Detection of Suicide Risk on Social Media" conducted by the kubapok team. We experimented with the following configurations of transformer-based models: fine-tuned DeBERTa, GPT-4o with CoT and few-shot prompting, and fine-tuned GPT-4o. The task setup was to classify social media posts into four categories: indicator, ideation, behavior, and attempt. Our findings demonstrate that the fine-tuned GPT-4o model outperforms two other configurations, achieving high accuracy in identifying suicide risk. Notably, our model achieved second place in the competition. By demonstrating that straightforward, general-purpose models can achieve state-of-the-art results, we propose that these models, combined with minimal tuning, may have the potential to be effective solutions for automated suicide risk detection on social media.