Graded Relevance Scoring of Written Essays with Dense Retrieval

TL;DR

This paper introduces a novel dense retrieval-based method for graded relevance scoring of essays, focusing on the relevance trait, achieving state-of-the-art results and reducing labeling costs.

Contribution

It proposes a new unsupervised dense retrieval approach using clustering and 1-NN classification for relevance scoring in essays, with strong performance on the ASAP++ dataset.

Findings

Achieved state-of-the-art results in task-specific relevance scoring.

Performed comparably to the best models in cross-task scenarios.

Reduced labeling effort by 90% with minimal performance loss.

Abstract

Automated Essay Scoring automates the grading process of essays, providing a great advantage for improving the writing proficiency of students. While holistic essay scoring research is prevalent, a noticeable gap exists in scoring essays for specific quality traits. In this work, we focus on the relevance trait, which measures the ability of the student to stay on-topic throughout the entire essay. We propose a novel approach for graded relevance scoring of written essays that employs dense retrieval encoders. Dense representations of essays at different relevance levels then form clusters in the embeddings space, such that their centroids are potentially separate enough to effectively represent their relevance levels. We hence use the simple 1-Nearest-Neighbor classification over those centroids to determine the relevance level of an unseen essay. As an effective unsupervised dense encoder, we leverage Contriever, which is pre-trained with contrastive learning and demonstrated comparable performance to supervised dense retrieval models. We tested our approach on both task-specific (i.e., training and testing on same task) and cross-task (i.e., testing on unseen task) scenarios using the widely used ASAP++ dataset. Our method establishes a new state-of-the-art performance in the task-specific scenario, while its extension for the cross-task scenario exhibited a performance that is on par with the state-of-the-art model for that scenario. We also analyzed the performance of our approach in a more practical few-shot scenario, showing that it can significantly reduce the labeling cost while sacrificing only 10% of its effectiveness.