Scaling Equitable Reflection Assessment in Education via Large Language Models and Role-Based Feedback Agents

TL;DR

This paper introduces a multi-agent LLM system that provides scalable, equitable formative feedback in education by combining role-based agents for scoring, bias checking, metacognitive prompting, and feedback generation, improving support for diverse learners.

Contribution

The paper presents a novel multi-agent LLM framework with role-based agents that produce equitable, high-quality formative feedback at scale, addressing resource limitations in education.

Findings

System achieves expert-level rubric scoring accuracy.

AI-generated comments are rated as helpful and empathetic.

Fairness checks help monitor and reduce scoring disparities.

Abstract

Formative feedback is widely recognized as one of the most effective drivers of student learning, yet it remains difficult to implement equitably at scale. In large or low-resource courses, instructors often lack the time, staffing, and bandwidth required to review and respond to every student reflection, creating gaps in support precisely where learners would benefit most. This paper presents a theory-grounded system that uses five coordinated role-based LLM agents (Evaluator, Equity Monitor, Metacognitive Coach, Aggregator, and Reflexion Reviewer) to score learner reflections with a shared rubric and to generate short, bias-aware, learner-facing comments. The agents first produce structured rubric scores, then check for potentially biased or exclusionary language, add metacognitive prompts that invite students to think about their own thinking, and finally compose a concise feedback message of at most 120 words. The system includes simple fairness checks that compare scoring error across lower and higher scoring learners, enabling instructors to monitor and bound disparities in accuracy. We evaluate the pipeline in a 12-session AI literacy program with adult learners. In this setting, the system produces rubric scores that approach expert-level agreement, and trained graders rate the AI-generated comments as helpful, empathetic, and well aligned with instructional goals. Taken together, these results show that multi-agent LLM systems can deliver equitable, high-quality formative feedback at a scale and speed that would be impossible for human graders alone. More broadly, the work points toward a future where feedback-rich learning becomes feasible for any course size or context, advancing long-standing goals of equity, access, and instructional capacity in education.