Zero-shot Large Language Models for Automatic Readability Assessment

TL;DR

This paper introduces a zero-shot prompting approach using large language models for automatic readability assessment, outperforming previous methods across diverse datasets and languages.

Contribution

It presents a novel zero-shot prompting methodology for ARA and introduces LAURAE, combining LLM and readability formulas for improved robustness.

Findings

Prompting methodology outperforms prior methods on 13 of 14 datasets.

LAURAE combines LLM and formula scores, enhancing robustness across languages and text types.

LLMs can be effectively used for unsupervised readability assessment without fine-tuning.

Abstract

Unsupervised automatic readability assessment (ARA) methods have important practical and research applications (e.g., ensuring medical or educational materials are suitable for their target audiences). In this paper, we propose a new zero-shot prompting methodology for ARA and present the first comprehensive evaluation of using large language models (LLMs) as an unsupervised ARA method by testing 10 diverse open-source LLMs (e.g., different sizes and developers) on 14 diverse datasets (e.g., different text lengths and languages). Our findings show that our proposed prompting methodology outperforms prior methods on 13 of the 14 datasets. Furthermore, we propose LAURAE, which combines LLM and readability formula scores to improve robustness by capturing both contextual and shallow (e.g., sentence length) features of readability. Our evaluation demonstrates that LAURAE robustly outperforms prior methods across languages, text lengths, and amounts of technical language.