Transformer-based Joint Modelling for Automatic Essay Scoring and Off-Topic Detection

TL;DR

This paper introduces an unsupervised transformer-based model that jointly scores essays and detects off-topic responses, improving accuracy and robustness in automated essay scoring systems.

Contribution

It proposes a novel joint model with a topic regularization module and hybrid loss, enhancing off-topic detection and scoring accuracy over existing methods.

Findings

Outperforms baseline and conventional methods on two datasets

Effective in detecting off-topic and adversarial essays

Robust against human-level perturbations

Abstract

Automated Essay Scoring (AES) systems are widely popular in the market as they constitute a cost-effective and time-effective option for grading systems. Nevertheless, many studies have demonstrated that the AES system fails to assign lower grades to irrelevant responses. Thus, detecting the off-topic response in automated essay scoring is crucial in practical tasks where candidates write unrelated text responses to the given task in the question. In this paper, we are proposing an unsupervised technique that jointly scores essays and detects off-topic essays. The proposed Automated Open Essay Scoring (AOES) model uses a novel topic regularization module (TRM), which can be attached on top of a transformer model, and is trained using a proposed hybrid loss function. After training, the AOES model is further used to calculate the Mahalanobis distance score for off-topic essay detection. Our proposed method outperforms the baseline we created and earlier conventional methods on two essay-scoring datasets in off-topic detection as well as on-topic scoring. Experimental evaluation results on different adversarial strategies also show how the suggested method is robust for detecting possible human-level perturbations.