How Students Use AI Feedback Matters: Experimental Evidence on Physics Achievement and Autonomy

TL;DR

This study investigates how different usage patterns of AI feedback influence high school students' physics achievement and autonomy, revealing that tailored AI interventions benefit high achievers but may hinder lower achievers' autonomy.

Contribution

It provides experimental evidence on the heterogeneous effects of AI feedback based on usage patterns and student achievement levels in high school physics education.

Findings

Low-achieving students improved with heuristic hints

High-achieving students' performance increased with autonomous help

Lower achievers' autonomy declined with on-demand AI interventions

Abstract

Despite the precision and adaptiveness of generative AI (GAI)-powered feedback provided to students, existing practice and literature might ignore how usage patterns impact student learning. This study examines the heterogeneous effects of GAI-powered personalized feedback on high school students' physics achievement and autonomy through two randomized controlled trials, with a major focus on usage patterns. Each experiment lasted for five weeks, involving a total of 387 students. Experiment 1 (n = 121) assessed compulsory usage of the personalized recommendation system, revealing that low-achieving students significantly improved academic performance (d = 0.673, p < 0.05) when receiving AI-generated heuristic solution hints, whereas medium-achieving students' performance declined (d = -0.539, p < 0.05) with conventional answers provided by workbook. Notably, high-achieving students experienced a significant decline in self-regulated learning (d = -0.477, p < 0.05) without any significant gains in achievement. Experiment 2 (n = 266) investigated the usage pattern of autonomous on-demand help, demonstrating that fully learner-controlled AI feedback significantly enhanced academic performance for high-achieving students (d = 0.378, p < 0.05) without negatively impacting their autonomy. However, autonomy notably declined among lower achievers exposed to on-demand AI interventions (d = -0.383, p < 0.05), particularly in the technical-psychological dimension (d = -0.549, p < 0.05), which has a large overlap with self-regulation. These findings underscore the importance of usage patterns when applying GAI-powered personalized feedback to students.