MeDiSumQA: Patient-Oriented Question-Answer Generation from Discharge Letters

TL;DR

This paper introduces MeDiSumQA, a new dataset for evaluating large language models in generating patient-friendly answers from discharge summaries, aiming to improve medical communication and patient understanding.

Contribution

The creation of MeDiSumQA dataset from discharge summaries and its use to evaluate LLMs for patient-oriented question-answering is a novel contribution.

Findings

General-purpose LLMs outperform biomedical models in this task.

Automated metrics show good correlation with human judgment.

Releasing the dataset aims to foster further research in patient-centered medical AI.

Abstract

While increasing patients' access to medical documents improves medical care, this benefit is limited by varying health literacy levels and complex medical terminology. Large language models (LLMs) offer solutions by simplifying medical information. However, evaluating LLMs for safe and patient-friendly text generation is difficult due to the lack of standardized evaluation resources. To fill this gap, we developed MeDiSumQA. MeDiSumQA is a dataset created from MIMIC-IV discharge summaries through an automated pipeline combining LLM-based question-answer generation with manual quality checks. We use this dataset to evaluate various LLMs on patient-oriented question-answering. Our findings reveal that general-purpose LLMs frequently surpass biomedical-adapted models, while automated metrics correlate with human judgment. By releasing MeDiSumQA on PhysioNet, we aim to advance the development of LLMs to enhance patient understanding and ultimately improve care outcomes.