OMIND: Framework for Knowledge Grounded Finetuning and Multi-Turn Dialogue Benchmark for Mental Health LLMs

TL;DR

This paper introduces the oMind framework for knowledge-grounded fine-tuning of mental health LLMs and presents a new multi-turn dialogue benchmark, significantly improving reasoning and conversational capabilities in mental health applications.

Contribution

The paper develops the oMind framework, including a large high-quality dataset and a novel benchmark for multi-turn mental health dialogues, addressing key challenges in domain adaptation.

Findings

oMind LLMs outperform baselines in core capabilities.

Significant improvement in reasoning with up to 80% win rate.

Introduction of oMind-Chat benchmark for multi-turn mental health dialogues.

Abstract

Large Language Models (LLMs) have shown remarkable capabilities for complex tasks, yet adaptation in medical domain, specifically mental health, poses specific challenges. Mental health is a rising concern globally with LLMs having large potential to help address the same. We highlight three primary challenges for LLMs in mental health - lack of high quality interpretable and knowledge grounded training data; training paradigms restricted to core capabilities, and evaluation of multi turn dialogue settings. Addressing it, we present oMind framework which includes training and aligning LLM agents for diverse capabilities including conversations; high quality ~164k multi-task SFT dataset, as a result of our generation pipeline based on Structured Knowledge retrieval, LLM based pruning, and review actions. We also introduce oMind-Chat - a novel multi turn benchmark dataset with expert annotated turn level and conversation level rubrics. Our diverse experiments on both core capabilities and conversations shows oMind LLMs consistently outperform baselines. oMind-LLM also shows significantly better reasoning with up to 80% win rate.