Evaluating the Evaluators: Are readability metrics good measures of readability?

TL;DR

This paper critically evaluates traditional readability metrics in Plain Language Summarization, demonstrating that language models better align with human judgments and suggesting improved evaluation practices.

Contribution

It reveals the poor correlation of traditional readability metrics with human judgments and introduces language models as superior evaluators in PLS.

Findings

Most traditional metrics correlate poorly with human judgments.

Language models achieve higher correlation with human readability assessments.

LMs better capture deeper readability aspects like background knowledge.

Abstract

Plain Language Summarization (PLS) aims to distill complex documents into accessible summaries for non-expert audiences. In this paper, we conduct a thorough survey of PLS literature, and identify that the current standard practice for readability evaluation is to use traditional readability metrics, such as Flesch-Kincaid Grade Level (FKGL). However, despite proven utility in other fields, these metrics have not been compared to human readability judgments in PLS. We evaluate 8 readability metrics and show that most correlate poorly with human judgments, including the most popular metric, FKGL. We then show that Language Models (LMs) are better judges of readability, with the best-performing model achieving a Pearson correlation of 0.56 with human judgments. Extending our analysis to PLS datasets, which contain summaries aimed at non-expert audiences, we find that LMs better capture deeper measures of readability, such as required background knowledge, and lead to different conclusions than the traditional metrics. Based on these findings, we offer recommendations for best practices in the evaluation of plain language summaries. We release our analysis code and survey data.