Supporters and Skeptics: LLM-based Analysis of Engagement with Mental Health (Mis)Information Content on Video-sharing Platforms

TL;DR

This study analyzes engagement with mental health misinformation on YouTube Shorts and Bitchute, introducing a new dataset and revealing linguistic patterns and audience susceptibility that could worsen stigma and misinformation spread.

Contribution

It presents the first quantitative analysis of mental health misinformation on video platforms, introduces the MentalMisinfo dataset, and evaluates LLMs for misinformation detection.

Findings

Few-shot LLMs effectively detect MHMisinfo videos.

Distinct linguistic patterns in comments reveal engagement with misinformation.

Comments may reinforce stigma and susceptibility to misinformation.

Abstract

Over one in five adults in the US lives with a mental illness. In the face of a shortage of mental health professionals and offline resources, online short-form video content has grown to serve as a crucial conduit for disseminating mental health help and resources. However, the ease of content creation and access also contributes to the spread of misinformation, posing risks to accurate diagnosis and treatment. Detecting and understanding engagement with such content is crucial to mitigating their harmful effects on public health. We perform the first quantitative study of the phenomenon using YouTube Shorts and Bitchute as the sites of study. We contribute MentalMisinfo, a novel labeled mental health misinformation (MHMisinfo) dataset of 739 videos (639 from Youtube and 100 from Bitchute) and 135372 comments in total, using an expert-driven annotation schema. We first found that few-shot in-context learning with large language models (LLMs) are effective in detecting MHMisinfo videos. Next, we discover distinct and potentially alarming linguistic patterns in how audiences engage with MHMisinfo videos through commentary on both video-sharing platforms. Across the two platforms, comments could exacerbate prevailing stigma with some groups showing heightened susceptibility to and alignment with MHMisinfo. We discuss technical and public health-driven adaptive solutions to tackling the "epidemic" of mental health misinformation online.