EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models

TL;DR

This paper introduces EssayJudge, a comprehensive multimodal benchmark designed to evaluate the essay scoring capabilities of Multimodal Large Language Models across various linguistic traits, addressing limitations of traditional AES methods.

Contribution

It presents the first multimodal benchmark for AES, leveraging MLLMs' strengths to evaluate lexical, sentence, and discourse traits without manual feature engineering.

Findings

MLLMs show gaps in discourse-level trait evaluation

EssayJudge reveals performance disparities between MLLMs and humans

Benchmark highlights areas for future MLLM improvements in AES

Abstract

Automated Essay Scoring (AES) plays a crucial role in educational assessment by providing scalable and consistent evaluations of writing tasks. However, traditional AES systems face three major challenges: (1) reliance on handcrafted features that limit generalizability, (2) difficulty in capturing fine-grained traits like coherence and argumentation, and (3) inability to handle multimodal contexts. In the era of Multimodal Large Language Models (MLLMs), we propose EssayJudge, the first multimodal benchmark to evaluate AES capabilities across lexical-, sentence-, and discourse-level traits. By leveraging MLLMs' strengths in trait-specific scoring and multimodal context understanding, EssayJudge aims to offer precise, context-rich evaluations without manual feature engineering, addressing longstanding AES limitations. Our experiments with 18 representative MLLMs reveal gaps in AES performance compared to human evaluation, particularly in discourse-level traits, highlighting the need for further advancements in MLLM-based AES research.