From Reddit to Generative AI: Evaluating Large Language Models for Anxiety Support Fine-tuned on Social Media Data

TL;DR

This paper systematically evaluates large language models like GPT and Llama for anxiety support, revealing that fine-tuning improves language but can increase toxicity and reduce emotional supportiveness, highlighting associated risks.

Contribution

It introduces a mixed-method evaluation framework for assessing LLMs in sensitive mental health domains and provides insights into the effects of social media data fine-tuning.

Findings

Fine-tuning improves linguistic quality but increases toxicity.

GPT is more supportive than Llama in evaluations.

Fine-tuning on social media data can reduce emotional responsiveness.

Abstract

The growing demand for accessible mental health support, compounded by workforce shortages and logistical barriers, has led to increased interest in utilizing Large Language Models (LLMs) for scalable and real-time assistance. However, their use in sensitive domains such as anxiety support remains underexamined. This study presents a systematic evaluation of LLMs (GPT and Llama) for their potential utility in anxiety support by using real user-generated posts from the r/Anxiety subreddit for both prompting and fine-tuning. Our approach utilizes a mixed-method evaluation framework incorporating three main categories of criteria: (i) linguistic quality, (ii) safety and trustworthiness, and (iii) supportiveness. Results show that fine-tuning LLMs with naturalistic anxiety-related data enhanced linguistic quality but increased toxicity and bias, and diminished emotional responsiveness. While LLMs exhibited limited empathy, GPT was evaluated as more supportive overall. Our findings highlight the risks of fine-tuning LLMs on unprocessed social media content without mitigation strategies.