Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation

TL;DR

This study evaluates the effectiveness of large language model-generated plain language summaries in healthcare, revealing that while they appear comparable to human summaries in subjective quality, they do not enhance understanding better, and automated metrics are unreliable.

Contribution

First large-scale evaluation comparing LLM-generated PLSs to human ones using both subjective ratings and comprehension tests, highlighting gaps in current automated evaluation metrics.

Findings

LLM-generated PLSs are perceived as similar to human summaries in subjective assessments.

Human-written PLSs significantly improve reader comprehension over LLM-generated ones.

Automated evaluation metrics do not correlate well with human judgments of PLS quality.

Abstract

Plain language summaries (PLSs) are essential for facilitating effective communication between clinicians and patients by making complex medical information easier for laypeople to understand and act upon. Large language models (LLMs) have recently shown promise in automating PLS generation, but their effectiveness in supporting health information comprehension remains unclear. Prior evaluations have generally relied on automated scores that do not measure understandability directly, or subjective Likert-scale ratings from convenience samples with limited generalizability. To address these gaps, we conducted a large-scale crowdsourced evaluation of LLM-generated PLSs using Amazon Mechanical Turk with 150 participants. We assessed PLS quality through subjective Likert-scale ratings focusing on simplicity, informativeness, coherence, and faithfulness; and objective multiple-choice comprehension and recall measures of reader understanding. Additionally, we examined the alignment between 10 automated evaluation metrics and human judgments. Our findings indicate that while LLMs can generate PLSs that appear indistinguishable from human-written ones in subjective evaluations, human-written PLSs lead to significantly better comprehension. Furthermore, automated evaluation metrics fail to reflect human judgment, calling into question their suitability for evaluating PLSs. This is the first study to systematically evaluate LLM-generated PLSs based on both reader preferences and comprehension outcomes. Our findings highlight the need for evaluation frameworks that move beyond surface-level quality and for generation methods that explicitly optimize for layperson comprehension.