Evaluating Document Simplification: On the Importance of Separately Assessing Simplicity and Meaning Preservation

TL;DR

This paper emphasizes the importance of separately evaluating simplification and meaning preservation in document-level text simplification, revealing biases in models and proposing reference-less metrics for better assessment.

Contribution

It introduces a framework for separately assessing simplification and meaning preservation, and evaluates existing models using these distinct metrics at the document level.

Findings

Models tend to favor either simplification or meaning preservation, rarely excelling at both.

Existing metrics can be effectively used without references for evaluation.

Document-level evaluation reveals biases not seen in sentence-level assessments.

Abstract

Text simplification intends to make a text easier to read while preserving its core meaning. Intuitively and as shown in previous works, these two dimensions (simplification and meaning preservation) are often-times inversely correlated. An overly conservative text will fail to simplify sufficiently, whereas extreme simplification will degrade meaning preservation. Yet, popular evaluation metrics either aggregate meaning preservation and simplification into a single score (SARI, LENS), or target meaning preservation alone (BERTScore, QuestEval). Moreover, these metrics usually require a set of references and most previous work has only focused on sentence-level simplification. In this paper, we focus on the evaluation of document-level text simplification and compare existing models using distinct metrics for meaning preservation and simplification. We leverage existing metrics from similar tasks and introduce a reference-less metric variant for simplicity, showing that models are mostly biased towards either simplification or meaning preservation, seldom performing well on both dimensions. Making use of the fact that the metrics we use are all reference-less, we also investigate the performance of existing models when applied to unseen data (where reference simplifications are unavailable).