Large Language Models as Partners in Student Essay Evaluation

TL;DR

This study explores the potential of Large Language Models to evaluate student essays, demonstrating their ability to match faculty assessments and proposing their role as collaborative partners in evaluation processes.

Contribution

It introduces a novel application of LLMs in essay evaluation, comparing different scenarios and analyzing their assessment quality and diversity.

Findings

Strong correlation between LLM and faculty assessments in pairwise comparison with rubrics

LLMs can match faculty assessment capabilities

Variations in LLM assessments reflect diversity, not confusion

Abstract

As the importance of comprehensive evaluation in workshop courses increases, there is a growing demand for efficient and fair assessment methods that reduce the workload for faculty members. This paper presents an evaluation conducted with Large Language Models (LLMs) using actual student essays in three scenarios: 1) without providing guidance such as rubrics, 2) with pre-specified rubrics, and 3) through pairwise comparison of essays. Quantitative analysis of the results revealed a strong correlation between LLM and faculty member assessments in the pairwise comparison scenario with pre-specified rubrics, although concerns about the quality and stability of evaluations remained. Therefore, we conducted a qualitative analysis of LLM assessment comments, showing that: 1) LLMs can match the assessment capabilities of faculty members, 2) variations in LLM assessments should be interpreted as diversity rather than confusion, and 3) assessments by humans and LLMs can differ and complement each other. In conclusion, this paper suggests that LLMs should not be seen merely as assistants to faculty members but as partners in evaluation committees and outlines directions for further research.