"My Grade is Wrong!": A Contestable AI Framework for Interactive Feedback in Evaluating Student Essays

TL;DR

This paper presents CAELF, a multi-agent framework using LLMs and argumentation to automate and improve interactive feedback for student essays, making it more feasible and effective in education.

Contribution

Introduces CAELF, a novel multi-agent, contestable AI framework that enables interactive feedback and reasoning in automated essay evaluation.

Findings

CAELF significantly improves feedback quality and interaction.

Enhanced reasoning capabilities of LLMs in educational feedback.

Effective in a case study with 500 critical thinking essays.

Abstract

Interactive feedback, where feedback flows in both directions between teacher and student, is more effective than traditional one-way feedback. However, it is often too time-consuming for widespread use in educational practice. While Large Language Models (LLMs) have potential for automating feedback, they struggle with reasoning and interaction in an interactive setting. This paper introduces CAELF, a Contestable AI Empowered LLM Framework for automating interactive feedback. CAELF allows students to query, challenge, and clarify their feedback by integrating a multi-agent system with computational argumentation. Essays are first assessed by multiple Teaching-Assistant Agents (TA Agents), and then a Teacher Agent aggregates the evaluations through formal reasoning to generate feedback and grades. Students can further engage with the feedback to refine their understanding. A case study on 500 critical thinking essays with user studies demonstrates that CAELF significantly improves interactive feedback, enhancing the reasoning and interaction capabilities of LLMs. This approach offers a promising solution to overcoming the time and resource barriers that have limited the adoption of interactive feedback in educational settings.