Empirical Analysis of the Effect of Context in the Task of Automated Essay Scoring in Transformer-Based Models

TL;DR

This paper investigates how incorporating various contextual factors into transformer-based models improves automated essay scoring, achieving higher accuracy and demonstrating the value of contextual enrichment over architecture modifications.

Contribution

It introduces a contextual augmentation approach for transformer-based AES models, significantly enhancing their performance across multiple essay sets.

Findings

Achieved a mean Quadratic Weighted Kappa score of 0.823 on the full dataset.

Outperformed prior transformer models in most essay sets.

Only 3.83% below the state-of-the-art deep learning model trained per essay set.

Abstract

Automated Essay Scoring (AES) has emerged to prominence in response to the growing demand for educational automation. Providing an objective and cost-effective solution, AES standardises the assessment of extended responses. Although substantial research has been conducted in this domain, recent investigations reveal that alternative deep-learning architectures outperform transformer-based models. Despite the successful dominance in the performance of the transformer architectures across various other tasks, this discrepancy has prompted a need to enrich transformer-based AES models through contextual enrichment. This study delves into diverse contextual factors using the ASAP-AES dataset, analysing their impact on transformer-based model performance. Our most effective model, augmented with multiple contextual dimensions, achieves a mean Quadratic Weighted Kappa score of 0.823 across the entire essay dataset and 0.8697 when trained on individual essay sets. Evidently surpassing prior transformer-based models, this augmented approach only underperforms relative to the state-of-the-art deep learning model trained essay-set-wise by an average of 3.83\% while exhibiting superior performance in three of the eight sets. Importantly, this enhancement is orthogonal to architecture-based advancements and seamlessly adaptable to any AES model. Consequently, this contextual augmentation methodology presents a versatile technique for refining AES capabilities, contributing to automated grading and evaluation evolution in educational settings.