Evaluating Small Decoder-Only Language Models for Grammar Correction and Text Simplification

TL;DR

This paper evaluates the potential of small decoder-only language models as efficient alternatives for grammar correction and text simplification, finding they currently underperform compared to larger models.

Contribution

It provides an empirical assessment of small language models' capabilities on rewriting tasks, highlighting their limitations and the need for further development.

Findings

Small models learn some behaviors well

Performance remains below strong baselines

Struggle with meaning retention and hallucinations

Abstract

Large language models have become extremely popular recently due to their ability to achieve strong performance on a variety of tasks, such as text generation and rewriting, but their size and computation cost make them difficult to access, deploy, and secure in many settings. This paper investigates whether small, decoder-only language models can provide an efficient alternative for the tasks of grammar correction and text simplification. The experiments in this paper focus on testing small language models out of the box, fine-tuned, and run sequentially on the JFLEG and ASSET datasets using established metrics. The results show that while SLMs may learn certain behaviors well, their performance remains below strong baselines and current LLMs. The results also show that SLMs struggle with retaining meaning and hallucinations. These findings suggest that despite their efficiency advantages, current SLMs are not yet competitive enough with modern LLMs for rewriting, and further advances in training are required for SLMs to close the performance gap between them and today's LLMs.