Automated Essay Scoring Using Transformer Models

TL;DR

This paper demonstrates that transformer models outperform traditional bag-of-words methods in automated essay scoring, offering more accurate and context-aware assessments, especially for tasks like politeness classification.

Contribution

The paper introduces a transformer-based approach for AES, compares it with BOW models, and provides implementation guidance, highlighting its advantages in accuracy and context understanding.

Findings

Transformer models outperform BOW models in AES tasks.

Transformer models improve the accuracy of human raters.

Detailed implementation instructions are provided for transformer-based AES.

Abstract

Automated essay scoring (AES) is gaining increasing attention in the education sector as it significantly reduces the burden of manual scoring and allows ad hoc feedback for learners. Natural language processing based on machine learning has been shown to be particularly suitable for text classification and AES. While many machine-learning approaches for AES still rely on a bag-of-words (BOW) approach, we consider a transformer-based approach in this paper, compare its performance to a logistic regression model based on the BOW approach and discuss their differences. The analysis is based on 2,088 email responses to a problem-solving task, that were manually labeled in terms of politeness. Both transformer models considered in that analysis outperformed without any hyper-parameter tuning the regression-based model. We argue that for AES tasks such as politeness classification, the transformer-based approach has significant advantages, while a BOW approach suffers from not taking word order into account and reducing the words to their stem. Further, we show how such models can help increase the accuracy of human raters, and we provide a detailed instruction on how to implement transformer-based models for one's own purpose.