Resource for Error Analysis in Text Simplification: New Taxonomy and Test Collection

TL;DR

This paper introduces a new error taxonomy and a test collection for evaluating errors in automatic text simplification, aiming to improve the assessment and development of more reliable simplification models.

Contribution

It presents a novel error taxonomy, a human-annotated dataset of simplified scientific texts, and an analysis of model performance in error detection and classification.

Findings

Existing metrics do not correlate well with error presence

The dataset enables better evaluation of simplification errors

Models show varying effectiveness in error detection and classification

Abstract

The general public often encounters complex texts but does not have the time or expertise to fully understand them, leading to the spread of misinformation. Automatic Text Simplification (ATS) helps make information more accessible, but its evaluation methods have not kept up with advances in text generation, especially with Large Language Models (LLMs). In particular, recent studies have shown that current ATS metrics do not correlate with the presence of errors. Manual inspections have further revealed a variety of errors, underscoring the need for a more nuanced evaluation framework, which is currently lacking. This resource paper addresses this gap by introducing a test collection for detecting and classifying errors in simplified texts. First, we propose a taxonomy of errors, with a formal focus on information distortion. Next, we introduce a parallel dataset of automatically simplified scientific texts. This dataset has been human-annotated with labels based on our proposed taxonomy. Finally, we analyze the quality of the dataset, and we study the performance of existing models to detect and classify errors from that taxonomy. These contributions give researchers the tools to better evaluate errors in ATS, develop more reliable models, and ultimately improve the quality of automatically simplified texts.