LLM-Generated Feedback Supports Learning If Learners Choose to Use It

TL;DR

This study examines how on-demand LLM-generated explanatory feedback influences learning in tutor training, finding moderate benefits when learners choose to use it, without increasing task completion time.

Contribution

It provides empirical evidence that learner engagement with LLM feedback enhances learning outcomes and offers open datasets and tools for reproducibility.

Findings

Learners who engaged with LLM feedback scored higher at posttest.

Two lessons showed statistically significant learning benefits from LLM feedback.

LLM feedback was rated as helpful and did not increase completion time.

Abstract

Large language models (LLMs) are increasingly used to generate feedback, yet their impact on learning remains underexplored, especially compared to existing feedback methods. This study investigates how on-demand LLM-generated explanatory feedback influences learning in seven scenario-based tutor training lessons. Analyzing over 2,600 lesson completions from 885 tutor learners, we compare posttest performance among learners across three groups: learners who received feedback generated by gpt-3.5-turbo, those who declined it, and those without access. All groups received non-LLM corrective feedback. To address potential selection bias-where higher-performing learners may be more inclined to use LLM feedback-we applied propensity scoring. Learners with a higher predicted likelihood of engaging with LLM feedback scored significantly higher at posttest than those with lower propensity. After adjusting for this effect, two out of seven lessons showed statistically significant learning benefits from LLM feedback with standardized effect sizes of 0.28 and 0.33. These moderate effects suggest that the effectiveness of LLM feedback depends on the learners' tendency to seek support. Importantly, LLM feedback did not significantly increase completion time, and learners overwhelmingly rated it as helpful. These findings highlight LLM feedback's potential as a low-cost and scalable way to improve learning on open-ended tasks, particularly in existing systems already providing feedback without LLMs. This work contributes open datasets, LLM prompts, and rubrics to support reproducibility.