Readability Formulas, Systems and LLMs are Poor Predictors of Reading Ease

TL;DR

This study evaluates traditional and modern readability scoring methods using real-time eye tracking data, revealing their poor predictive power for actual reading ease across diverse readers and texts, and emphasizing the need for cognitively grounded approaches.

Contribution

Introduces a novel evaluation framework for readability methods based on real-time eye tracking data, demonstrating the inadequacy of existing models and highlighting the potential of psycholinguistic features.

Findings

Traditional and ML-based readability formulas perform poorly in predicting real-time reading ease.

Psycholinguistic word properties often outperform existing readability measures.

Current methods are ineffective across different reader groups and text lengths.

Abstract

Methods for scoring text readability have been studied for over a century, and are widely used in research and in user-facing applications in many domains. Thus far, the development and evaluation of such methods have primarily relied on two types of offline behavioral data, performance on reading comprehension tests and ratings of text readability levels. In this work, we instead focus on a fundamental and understudied aspect of readability, real-time reading ease, captured with online reading measures using eye tracking. We introduce an evaluation framework for readability scoring methods which quantifies their ability to account for reading ease, while controlling for content variation across texts. Applying this evaluation to prominent traditional readability formulas, modern machine learning systems, frontier Large Language Models and commercial systems used in education, suggests that they are all poor predictors of reading ease in English. This outcome holds across native and non-native speakers, reading regimes, and textual units of different lengths. The evaluation further reveals that existing methods are often outperformed by word properties commonly used in psycholinguistics for prediction of reading times. Our results highlight a fundamental limitation of existing approaches to readability scoring, the utility of psycholinguistics for readability research, and the need for new, cognitively driven readability scoring approaches that can better account for reading ease.