Automated Feedback Loops to Protect Text Simplification with Generative AI from Information Loss

TL;DR

This paper investigates methods to detect and recover missing information in health-related text simplified by generative AI, demonstrating that adding all missing entities enhances the quality of the simplified text.

Contribution

It introduces a comparative approach to identify and insert missing information in AI-simplified health texts, highlighting the effectiveness of adding all missing entities for better content preservation.

Findings

Adding all missing entities improves text quality.

Entity addition outperforms top-ranked or random entity insertion.

Current tools can identify entities but struggle with ranking them.

Abstract

Understanding health information is essential in achieving and maintaining a healthy life. We focus on simplifying health information for better understanding. With the availability of generative AI, the simplification process has become efficient and of reasonable quality, however, the algorithms remove information that may be crucial for comprehension. In this study, we compare generative AI to detect missing information in simplified text, evaluate its importance, and fix the text with the missing information. We collected 50 health information texts and simplified them using gpt-4-0613. We compare five approaches to identify missing elements and regenerate the text by inserting the missing elements. These five approaches involve adding missing entities and missing words in various ways: 1) adding all the missing entities, 2) adding all missing words, 3) adding the top-3 entities ranked by gpt-4-0613, and 4, 5) serving as controls for comparison, adding randomly chosen entities. We use cosine similarity and ROUGE scores to evaluate the semantic similarity and content overlap between the original, simplified, and reconstructed simplified text. We do this for both summaries and full text. Overall, we find that adding missing entities improves the text. Adding all the missing entities resulted in better text regeneration, which was better than adding the top-ranked entities or words, or random words. Current tools can identify these entities, but are not valuable in ranking them.