Text and Audio Simplification: Human vs. ChatGPT

TL;DR

This study compares human and ChatGPT methods for simplifying text and audio in healthcare, evaluating their effectiveness with multiple metrics and expert opinions to understand their impact on comprehension and content retention.

Contribution

It provides a systematic comparison of ChatGPT and human simplification across multiple metrics and includes expert evaluation in the medical domain.

Findings

ChatGPT simplifies texts effectively, aligning with human simplification in many metrics.

Simple corpora are more similar to human simplified texts.

Medical experts prefer ChatGPT's style but find content retention lower.

Abstract

Text and audio simplification to increase information comprehension are important in healthcare. With the introduction of ChatGPT, an evaluation of its simplification performance is needed. We provide a systematic comparison of human and ChatGPT simplified texts using fourteen metrics indicative of text difficulty. We briefly introduce our online editor where these simplification tools, including ChatGPT, are available. We scored twelve corpora using our metrics: six text, one audio, and five ChatGPT simplified corpora. We then compare these corpora with texts simplified and verified in a prior user study. Finally, a medical domain expert evaluated these texts and five, new ChatGPT simplified versions. We found that simple corpora show higher similarity with the human simplified texts. ChatGPT simplification moves metrics in the right direction. The medical domain expert evaluation showed a preference for the ChatGPT style, but the text itself was rated lower for content retention.