Neural Automated Essay Scoring and Coherence Modeling for Adversarially Crafted Input

TL;DR

This paper introduces a neural local coherence model integrated with AES to detect adversarial, incoherent essays, improving robustness and validity of automated scoring systems.

Contribution

It presents a novel neural coherence model and a joint training framework that enhances AES models' ability to identify incoherent, adversarial essay inputs.

Findings

Improved detection of incoherent, adversarial essays

Enhanced AES robustness against adversarial input

Effective joint training of coherence and scoring models

Abstract

We demonstrate that current state-of-the-art approaches to Automated Essay Scoring (AES) are not well-suited to capturing adversarially crafted input of grammatical but incoherent sequences of sentences. We develop a neural model of local coherence that can effectively learn connectedness features between sentences, and propose a framework for integrating and jointly training the local coherence model with a state-of-the-art AES model. We evaluate our approach against a number of baselines and experimentally demonstrate its effectiveness on both the AES task and the task of flagging adversarial input, further contributing to the development of an approach that strengthens the validity of neural essay scoring models.