Prompt Agnostic Essay Scorer: A Domain Generalization Approach to Cross-prompt Automated Essay Scoring

TL;DR

This paper introduces PAES, a prompt-agnostic essay scoring model that performs well across prompts without needing prompt-specific data, advancing the practicality of automated essay scoring systems.

Contribution

The paper presents PAES, a novel single-stage, prompt-agnostic model for cross-prompt AES that does not require labeled or unlabeled target-prompt data during training.

Findings

Achieves state-of-the-art performance on the ASAP dataset.

Does not require prompt-specific data for training.

Simplifies cross-prompt AES application.

Abstract

Cross-prompt automated essay scoring (AES) requires the system to use non target-prompt essays to award scores to a target-prompt essay. Since obtaining a large quantity of pre-graded essays to a particular prompt is often difficult and unrealistic, the task of cross-prompt AES is vital for the development of real-world AES systems, yet it remains an under-explored area of research. Models designed for prompt-specific AES rely heavily on prompt-specific knowledge and perform poorly in the cross-prompt setting, whereas current approaches to cross-prompt AES either require a certain quantity of labelled target-prompt essays or require a large quantity of unlabelled target-prompt essays to perform transfer learning in a multi-step manner. To address these issues, we introduce Prompt Agnostic Essay Scorer (PAES) for cross-prompt AES. Our method requires no access to labelled or unlabelled target-prompt data during training and is a single-stage approach. PAES is easy to apply in practice and achieves state-of-the-art performance on the Automated Student Assessment Prize (ASAP) dataset.