MHQA: A Diverse, Knowledge Intensive Mental Health Question Answering Challenge for Language Models

TL;DR

This paper introduces MHQA, a comprehensive benchmark dataset for mental health question answering, covering multiple domains and question types, to evaluate and improve large language models in healthcare contexts.

Contribution

The work presents the first diverse, expert-verified mental health QA dataset with multiple question types and a rigorous pipeline for dataset creation, filling a critical gap in mental health NLP benchmarking.

Findings

LLMs achieve varying F1 scores on MHQA, indicating room for improvement.

Few-shot and fine-tuning methods impact model performance differently.

The dataset enables detailed analysis of LLM capabilities in mental health QA.

Abstract

Mental health remains a challenging problem all over the world, with issues like depression, anxiety becoming increasingly common. Large Language Models (LLMs) have seen a vast application in healthcare, specifically in answering medical questions. However, there is a lack of standard benchmarking datasets for question answering (QA) in mental health. Our work presents a novel multiple choice dataset, MHQA (Mental Health Question Answering), for benchmarking Language models (LMs). Previous mental health datasets have focused primarily on text classification into specific labels or disorders. MHQA, on the other hand, presents question-answering for mental health focused on four key domains: anxiety, depression, trauma, and obsessive/compulsive issues, with diverse question types, namely, factoid, diagnostic, prognostic, and preventive. We use PubMed abstracts as the primary source for QA. We develop a rigorous pipeline for LLM-based identification of information from abstracts based on various selection criteria and converting it into QA pairs. Further, valid QA pairs are extracted based on post-hoc validation criteria. Overall, our MHQA dataset consists of 2,475 expert-verified gold standard instances called MHQA-gold and ~56.1k pairs pseudo labeled using external medical references. We report F1 scores on different LLMs along with few-shot and supervised fine-tuning experiments, further discussing the insights for the scores.