MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare

TL;DR

This paper introduces MentalBERT and MentalRoBERTa, two domain-specific pretrained language models designed for mental healthcare, demonstrating their effectiveness in improving mental disorder detection from social content.

Contribution

The paper presents the first pretrained language models specifically for mental healthcare, tailored to improve mental health detection tasks.

Findings

MentalBERT and MentalRoBERTa outperform general models on mental disorder benchmarks.

Pretraining in the mental health domain enhances detection accuracy.

Models are publicly available for research use.

Abstract

Mental health is a critical issue in modern society, and mental disorders could sometimes turn to suicidal ideation without adequate treatment. Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention. Recent advances in pretrained contextualized language representations have promoted the development of several domain-specific pretrained models and facilitated several downstream applications. However, there are no existing pretrained language models for mental healthcare. This paper trains and release two pretrained masked language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community. Besides, we evaluate our trained domain-specific models and several variants of pretrained language models on several mental disorder detection benchmarks and demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks.