Exploration of Summarization by Generative Language Models for Automated Scoring of Long Essays

TL;DR

This paper investigates using generative language models for automated scoring of long essays by leveraging summarization and prompting techniques, significantly improving scoring accuracy over traditional encoder-based models.

Contribution

It introduces a novel approach employing generative models for long essay scoring, overcoming token length limitations of previous methods.

Findings

QWK increased from 0.822 to 0.8878

Generative models outperform encoder-based models on long essays

Summarization and prompting enhance scoring accuracy

Abstract

BERT and its variants are extensively explored for automated scoring. However, a limit of 512 tokens for these encoder-based models showed the deficiency in automated scoring of long essays. Thus, this research explores generative language models for automated scoring of long essays via summarization and prompting. The results revealed great improvement of scoring accuracy with QWK increased from 0.822 to 0.8878 for the Learning Agency Lab Automated Essay Scoring 2.0 dataset.