Assessing the Impact and Underlying Pathways of Sequenced AI feedback on Student Learning

TL;DR

This study investigates how sequenced AI feedback affects student engagement and learning, revealing that while it increases engagement and perceived support, it can negatively impact actual learning outcomes.

Contribution

It provides empirical evidence that sequenced AI feedback enhances engagement but may impair learning performance, highlighting important design considerations for AI-driven educational tools.

Findings

Sequenced feedback increased behavioral engagement and perceived encouragement.

It induced higher mental effort and was perceived as more supportive.

Overall, sequenced feedback resulted in poorer learning outcomes.

Abstract

Feedback is essential for learning, but its effectiveness relies heavily on how well it engages students in the educational process. Generative AI offers novel opportunities to efficiently produce rich, formative feedback, ranging from direct explanations to incrementally sequenced scaffolding designed to promote learner autonomy. Despite these capabilities, it is still unclear whether sequenced (layered) AI feedback -- which provides encouragement and hints before revealing the correct answer -- genuinely enhances engagement and learning outcomes. To investigate this, we randomly assigned 199 participants to receive either sequenced or non-sequenced AI-generated feedback. We evaluated its impact on learning performance, cognitive and behavioral engagement, and affective perceptions to understand how these factors mediate overall learning outcomes. Results show that sequenced feedback elicited slightly higher behavioral engagement and, as anticipated, was perceived as more encouraging and supportive of student independence. Concurrently, however, it induced a higher level of mental effort. Mediation analyses identified a positive affective pathway driven by perceived encouragement, which was completely counteracted by a negative behavioral pathway associated with the average number of tasks requiring three or more submissions; the cognitive pathway (mental effort) remained non-significant. Overall, sequenced feedback led to significantly poorer learning outcomes when compared to direct, non-sequenced feedback. These findings highlight a crucial trade-off: although sequenced AI scaffolding boosts engagement and positive user perceptions, it can have a detrimental effect on actual learning performance. By integrating analyses of outcomes, perceptions, and underlying mechanisms, this study provides nuanced insights for designing automated, AI-driven feedback systems.