Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions

TL;DR

This study evaluates whether ChatGPT can deliver more empathetic responses than physicians in healthcare interactions, using novel metrics and real patient data, indicating potential for improved patient care and reduced burnout.

Contribution

It introduces a new empathy ranking evaluation (EMRank) combining automated and human assessments, and demonstrates ChatGPT's potential to outperform physicians in empathetic communication.

Findings

LLMs can surpass physicians in empathy levels

Proposed EMRank effectively measures empathy in responses

LLMs show promise for enhancing healthcare communication

Abstract

The integration of Large Language Models (LLMs) into the healthcare domain has the potential to significantly enhance patient care and support through the development of empathetic, patient-facing chatbots. This study investigates an intriguing question Can ChatGPT respond with a greater degree of empathy than those typically offered by physicians? To answer this question, we collect a de-identified dataset of patient messages and physician responses from Mayo Clinic and generate alternative replies using ChatGPT. Our analyses incorporate novel empathy ranking evaluation (EMRank) involving both automated metrics and human assessments to gauge the empathy level of responses. Our findings indicate that LLM-powered chatbots have the potential to surpass human physicians in delivering empathetic communication, suggesting a promising avenue for enhancing patient care and reducing professional burnout. The study not only highlights the importance of empathy in patient interactions but also proposes a set of effective automatic empathy ranking metrics, paving the way for the broader adoption of LLMs in healthcare.