Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading

TL;DR

This study compares multi-agent and single-agent large language models for automated essay scoring, revealing that multi-agent systems excel at identifying weak essays and that few-shot calibration significantly enhances performance.

Contribution

It introduces a multi-agent architecture with specialist agents for essay grading and compares it to a single-agent system, highlighting the impact of calibration and architectural choices.

Findings

Multi-agent systems better detect weak essays.

Single-agent systems perform well on mid-range essays.

Few-shot calibration improves scores by ~26%.

Abstract

Automated essay scoring (AES) systems increasingly rely on large language models, yet little is known about how architectural choices shape their performance across different essay quality levels. This paper evaluates single-agent and multi-agent LLM architectures for essay grading using the ASAP 2.0 corpus. Our multi-agent system decomposes grading into three specialist agents (Content, Structure, Language) coordinated by a Chairman Agent that implements rubric-aligned logic including veto rules and score capping. We test both architectures in zero-shot and few-shot conditions using GPT-5.1. Results show that the multi-agent system is significantly better at identifying weak essays while the single-agent system performs better on mid-range essays. Both architectures struggle with high-quality essays. Critically, few-shot calibration emerges as the dominant factor in system performance -- providing just two examples per score level improves QWK by approximately 26% for both architectures. These findings suggest architectural choice should align with specific deployment priorities, with multi-agent AI particularly suited for diagnostic screening of at-risk students, while single-agent models provide a cost-effective solution for general assessment.