CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring

TL;DR

CAFES is a novel collaborative multi-agent framework for multimodal essay scoring that improves alignment with human judgment and enhances evaluation accuracy, especially in grammatical and lexical aspects.

Contribution

It introduces the first multi-agent system for AES that combines rapid scoring, evidence aggregation, and iterative refinement to address multimodal evaluation challenges.

Findings

Achieved 21% improvement in Quadratic Weighted Kappa over baseline models.

Effectively enhances scoring accuracy for grammatical and lexical diversity.

Demonstrated robustness across multimodal assessments using state-of-the-art MLLMs.

Abstract

Automated Essay Scoring (AES) is crucial for modern education, particularly with the increasing prevalence of multimodal assessments. However, traditional AES methods struggle with evaluation generalizability and multimodal perception, while even recent Multimodal Large Language Model (MLLM)-based approaches can produce hallucinated justifications and scores misaligned with human judgment. To address the limitations, we introduce CAFES, the first collaborative multi-agent framework specifically designed for AES. It orchestrates three specialized agents: an Initial Scorer for rapid, trait-specific evaluations; a Feedback Pool Manager to aggregate detailed, evidence-grounded strengths; and a Reflective Scorer that iteratively refines scores based on this feedback to enhance human alignment. Extensive experiments, using state-of-the-art MLLMs, achieve an average relative improvement of 21% in Quadratic Weighted Kappa (QWK) against ground truth, especially for grammatical and lexical diversity. Our proposed CAFES framework paves the way for an intelligent multimodal AES system. The code will be available upon acceptance.