Unlocking LLMs: Addressing Scarce Data and Bias Challenges in Mental Health

TL;DR

This paper introduces IC-AnnoMI, a new expert-annotated dataset for motivational interviewing dialogues generated with LLMs, addressing challenges like bias and hallucinations in sensitive healthcare applications.

Contribution

The work presents a novel dataset and evaluation framework for LLMs in mental health, incorporating expert annotations and bias mitigation strategies in conversational therapy contexts.

Findings

IC-AnnoMI enhances LLM training with expert-annotated dialogues.

Prompting strategies influence bias reduction in LLM-generated therapy dialogues.

Augmented data improves LLM performance and reduces bias in sensitive domains.

Abstract

Large language models (LLMs) have shown promising capabilities in healthcare analysis but face several challenges like hallucinations, parroting, and bias manifestation. These challenges are exacerbated in complex, sensitive, and low-resource domains. Therefore, in this work we introduce IC-AnnoMI, an expert-annotated motivational interviewing (MI) dataset built upon AnnoMI by generating in-context conversational dialogues leveraging LLMs, particularly ChatGPT. IC-AnnoMI employs targeted prompts accurately engineered through cues and tailored information, taking into account therapy style (empathy, reflection), contextual relevance, and false semantic change. Subsequently, the dialogues are annotated by experts, strictly adhering to the Motivational Interviewing Skills Code (MISC), focusing on both the psychological and linguistic dimensions of MI dialogues. We comprehensively evaluate the IC-AnnoMI dataset and ChatGPT's emotional reasoning ability and understanding of domain intricacies by modeling novel classification tasks employing several classical machine learning and current state-of-the-art transformer approaches. Finally, we discuss the effects of progressive prompting strategies and the impact of augmented data in mitigating the biases manifested in IC-AnnoM. Our contributions provide the MI community with not only a comprehensive dataset but also valuable insights for using LLMs in empathetic text generation for conversational therapy in supervised settings.