Readability Measures and Automatic Text Simplification: In the Search of a Construct

TL;DR

This paper explores how traditional readability measures relate to automatic evaluation metrics and human judgment in automatic text simplification, highlighting the low correlations and the need for a clearer construct definition.

Contribution

It investigates the relationship between readability measures, automatic metrics, and human judgment in ATS, emphasizing the low correlation and the importance of defining the simplification construct.

Findings

Readability measures do not correlate well with automatic metrics.

Readability measures do not correlate well with human judgment.

Different evaluation angles for simplification show low correlation.

Abstract

Readability is a key concept in the current era of abundant written information. To help making texts more readable and make information more accessible to everyone, a line of researched aims at making texts accessible for their target audience: automatic text simplification (ATS). Lately, there have been studies on the correlations between automatic evaluation metrics in ATS and human judgment. However, the correlations between those two aspects and commonly available readability measures (such as readability formulas or linguistic features) have not been the focus of as much attention. In this work, we investigate the place of readability measures in ATS by complementing the existing studies on evaluation metrics and human judgment, on English. We first discuss the relationship between ATS and research in readability, then we report a study on correlations between readability measures and human judgment, and between readability measures and ATS evaluation metrics. We identify that in general, readability measures do not correlate well with automatic metrics and human judgment. We argue that as the three different angles from which simplification can be assessed tend to exhibit rather low correlations with one another, there is a need for a clear definition of the construct in ATS.