REVA: Supporting LLM-Generated Programming Feedback Validation at Scale Through User Attention-based Adaptation

TL;DR

REVA is a human-AI system that enhances the scalability and quality of reviewing AI-generated programming feedback by adaptively learning from instructor attention and propagating revisions across similar instances.

Contribution

The paper presents REVA, a novel adaptive system that leverages instructor attention to improve validation and revision of AI-generated feedback at scale.

Findings

REVA reduces instructor workload in feedback review.

REVA improves consistency of feedback revisions.

System is effective in educational settings.

Abstract

This paper introduces REVA, a human-AI system that expedites instructor review of voluminous AI-generated programming feedback by sequencing submissions to minimize cognitive context shifts and propagating instructor-driven revisions across semantically similar instances. REVA introduces a novel approach to human-AI collaboration in educational feedback by adaptively learning from instructors' attention in the review and revision process to continuously improve the feedback validation process. REVA's usefulness and effectiveness in improving feedback quality and the overall feedback review process were evaluated through a within-subjects lab study with 12 participants.