AI-Mediated Feedback Improves Student Revisions: A Randomized Trial with FeedbackWriter in a Large Undergraduate Course

TL;DR

This study demonstrates that AI-mediated feedback, integrated into the grading process, significantly enhances student revisions in a large undergraduate course, with effectiveness increasing as TAs utilize more AI suggestions.

Contribution

Introduces FeedbackWriter, a system that combines AI suggestions with TA feedback, improving student revision quality in large courses.

Findings

AI feedback led to higher-quality student revisions

TA adoption of AI suggestions correlated with improvement

TAs found AI suggestions helpful for identifying gaps

Abstract

Despite growing interest in using LLMs to generate feedback on students' writing, little is known about how students respond to AI-mediated versus human-provided feedback. We address this gap through a randomized controlled trial in a large introductory economics course (N=354), where we introduce and deploy FeedbackWriter - a system that generates AI suggestions to teaching assistants (TAs) while they provide feedback on students' knowledge-intensive essays. TAs have the full capacity to adopt, edit, or dismiss the suggestions. Students were randomly assigned to receive either handwritten feedback from TAs (baseline) or AI-mediated feedback where TAs received suggestions from FeedbackWriter. Students revise their drafts based on the feedback, which is further graded. In total, 1,366 essays were graded using the system. We found that students receiving AI-mediated feedback produced significantly higher-quality revisions, with gains increasing as TAs adopted more AI suggestions. TAs found the AI suggestions useful for spotting gaps and clarifying rubrics.