DS@GT eRisk 2024: Sentence Transformers for Social Media Risk Assessment

TL;DR

This paper explores the use of sentence transformers and classical machine learning models for social media risk assessment tasks related to depression and eating disorders, highlighting the importance of text representation.

Contribution

It introduces a ranking system for depression symptom prediction and demonstrates the effectiveness of sentence transformers in social media risk assessment tasks.

Findings

Binary classifiers perform poorly for ranking tasks.

Classical machine learning models are competitive with baselines.

Sentence transformers are effective for text representation.

Abstract

We present working notes for DS@GT team in the eRisk 2024 for Tasks 1 and 3. We propose a ranking system for Task 1 that predicts symptoms of depression based on the Beck Depression Inventory (BDI-II) questionnaire using binary classifiers trained on question relevancy as a proxy for ranking. We find that binary classifiers are not well calibrated for ranking, and perform poorly during evaluation. For Task 3, we use embeddings from BERT to predict the severity of eating disorder symptoms based on user post history. We find that classical machine learning models perform well on the task, and end up competitive with the baseline models. Representation of text data is crucial in both tasks, and we find that sentence transformers are a powerful tool for downstream modeling. Source code and models are available at \url{https://github.com/dsgt-kaggle-clef/erisk-2024}.