Automated Topical Component Extraction Using Neural Network Attention Scores from Source-based Essay Scoring

TL;DR

This paper introduces a method to extract Topical Components from source texts using neural network attention scores, enhancing automated essay scoring and providing better support for automated writing evaluation.

Contribution

It presents a novel approach linking AWE and neural AES by extracting evidence-based features from attention layers, improving interpretability and scoring performance.

Findings

Performance is comparable with manually or automatically constructed TCs.

Using TCs improves the interpretability of neural AES.

The method supports both feature extraction and essay grading.

Abstract

While automated essay scoring (AES) can reliably grade essays at scale, automated writing evaluation (AWE) additionally provides formative feedback to guide essay revision. However, a neural AES typically does not provide useful feature representations for supporting AWE. This paper presents a method for linking AWE and neural AES, by extracting Topical Components (TCs) representing evidence from a source text using the intermediate output of attention layers. We evaluate performance using a feature-based AES requiring TCs. Results show that performance is comparable whether using automatically or manually constructed TCs for 1) representing essays as rubric-based features, 2) grading essays.