LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing

TL;DR

This study demonstrates that large language models can effectively perform multi-dimensional analytic writing assessments, providing scores and comments comparable to human experts, with a scalable and reproducible evaluation framework.

Contribution

The paper introduces a novel, interpretable framework for evaluating LLM-generated feedback and demonstrates LLMs' capability in multi-criteria academic writing assessment.

Findings

LLMs produce reasonably good assessment scores

LLMs generate reliable feedback comments

The evaluation framework is scalable and reproducible

Abstract

The paper explores the performance of LLMs in the context of multi-dimensional analytic writing assessments, i.e. their ability to provide both scores and comments based on multiple assessment criteria. Using a corpus of literature reviews written by L2 graduate students and assessed by human experts against 9 analytic criteria, we prompt several popular LLMs to perform the same task under various conditions. To evaluate the quality of feedback comments, we apply a novel feedback comment quality evaluation framework. This framework is interpretable, cost-efficient, scalable, and reproducible, compared to existing methods that rely on manual judgments. We find that LLMs can generate reasonably good and generally reliable multi-dimensional analytic assessments. We release our corpus and code for reproducibility.