Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features

TL;DR

This paper enhances readability assessment by integrating novel linguistic features with transformer models, demonstrating that hybrid models outperform traditional and transformer-only approaches, especially on smaller datasets, achieving near-perfect accuracy.

Contribution

It introduces three new semantic features and shows that combining handcrafted features with transformers significantly improves readability classification performance.

Findings

Hybrid models outperform standalone models.

Handcrafted features boost performance on small datasets.

Achieved near-perfect accuracy of 99% on benchmark datasets.

Abstract

We report two essential improvements in readability assessment: 1. three novel features in advanced semantics and 2. the timely evidence that traditional ML models (e.g. Random Forest, using handcrafted features) can combine with transformers (e.g. RoBERTa) to augment model performance. First, we explore suitable transformers and traditional ML models. Then, we extract 255 handcrafted linguistic features using self-developed extraction software. Finally, we assemble those to create several hybrid models, achieving state-of-the-art (SOTA) accuracy on popular datasets in readability assessment. The use of handcrafted features help model performance on smaller datasets. Notably, our RoBERTA-RF-T1 hybrid achieves the near-perfect classification accuracy of 99%, a 20.3% increase from the previous SOTA.