Can Large Language Models Automatically Score Proficiency of Written Essays?

TL;DR

This study evaluates the ability of large language models, ChatGPT and Llama, to score written essays automatically, comparing their performance to state-of-the-art models and exploring their potential to provide useful feedback.

Contribution

The paper demonstrates that LLMs can effectively score essays and offer feedback, with performance influenced by prompt design and model choice, highlighting their potential in automated essay scoring.

Findings

LLMs show comparable average performance in essay scoring

Prompt choice significantly affects model performance

LLMs can provide useful feedback to improve essay quality

Abstract

Although several methods were proposed to address the problem of automated essay scoring (AES) in the last 50 years, there is still much to desire in terms of effectiveness. Large Language Models (LLMs) are transformer-based models that demonstrate extraordinary capabilities on various tasks. In this paper, we test the ability of LLMs, given their powerful linguistic knowledge, to analyze and effectively score written essays. We experimented with two popular LLMs, namely ChatGPT and Llama. We aim to check if these models can do this task and, if so, how their performance is positioned among the state-of-the-art (SOTA) models across two levels, holistically and per individual writing trait. We utilized prompt-engineering tactics in designing four different prompts to bring their maximum potential to this task. Our experiments conducted on the ASAP dataset revealed several interesting observations. First, choosing the right prompt depends highly on the model and nature of the task. Second, the two LLMs exhibited comparable average performance in AES, with a slight advantage for ChatGPT. Finally, despite the performance gap between the two LLMs and SOTA models in terms of predictions, they provide feedback to enhance the quality of the essays, which can potentially help both teachers and students.