EmplifAI: a Fine-grained Dataset for Japanese Empathetic Medical Dialogues in 28 Emotion Labels

TL;DR

This paper presents EmplifAI, a detailed Japanese empathetic dialogue dataset with 28 emotion labels, designed to improve emotional understanding and response in medical conversations, validated through model evaluation and human comparison.

Contribution

The creation of EmplifAI, a fine-grained, situation-based Japanese empathetic dialogue dataset with 28 emotion categories, and its use to evaluate and enhance LLMs in medical empathy tasks.

Findings

LLMs achieved an F1 score of 0.83 on emotional alignment.

Fine-tuning Japanese LLMs improved empathy and fluency.

Evaluation pipeline validated by human and LLM comparison.

Abstract

This paper introduces EmplifAI, a Japanese empathetic dialogue dataset designed to support patients coping with chronic medical conditions. They often experience a wide range of positive and negative emotions (e.g., hope and despair) that shift across different stages of disease management. EmplifAI addresses this complexity by providing situation-based dialogues grounded in 28 fine-grained emotion categories, adapted and validated from the GoEmotions taxonomy. The dataset includes 280 medically contextualized situations and 4125 two-turn dialogues, collected through crowdsourcing and expert review. To evaluate emotional alignment in empathetic dialogues, we assessed model predictions on situation--dialogue pairs using BERTScore across multiple large language models (LLMs), achieving F1 scores of 0.83. Fine-tuning a baseline Japanese LLM (LLM-jp-3.1-13b-instruct4) with EmplifAI resulted in notable improvements in fluency, general empathy, and emotion-specific empathy. Furthermore, we compared the scores assigned by LLM-as-a-Judge and human raters on dialogues generated by multiple LLMs to validate our evaluation pipeline and discuss the insights and potential risks derived from the correlation analysis.