Automated Essay Scoring Incorporating Annotations from Automated Feedback Systems
Christopher Ormerod

TL;DR
This paper enhances automated essay scoring by integrating feedback-driven annotations, such as grammatical errors and argumentative components, using large language models to improve scoring accuracy on the PERSUADE corpus.
Contribution
It introduces a novel method of incorporating feedback annotations from LLMs into AES scoring pipelines, improving accuracy over traditional methods.
Findings
Annotations improve scoring accuracy
LLMs effectively identify errors and argumentative elements
Enhanced scoring models outperform baseline approaches
Abstract
This study illustrates how incorporating feedback-oriented annotations into the scoring pipeline can enhance the accuracy of automated essay scoring (AES). This approach is demonstrated with the Persuasive Essays for Rating, Selecting, and Understanding Argumentative and Discourse Elements (PERSUADE) corpus. We integrate two types of feedback-driven annotations: those that identify spelling and grammatical errors, and those that highlight argumentative components. To illustrate how this method could be applied in real-world scenarios, we employ two LLMs to generate annotations -- a generative language model used for spell correction and an encoder-based token-classifier trained to identify and mark argumentative elements. By incorporating annotations into the scoring process, we demonstrate improvements in performance using encoder-based large language models fine-tuned as classifiers.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Sentiment Analysis and Opinion Mining · Speech and dialogue systems
