The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance

TL;DR

This study investigates how different structured LLM-generated feedback types influence programming students' problem-solving speed and effort, highlighting the importance of feedback design in educational outcomes.

Contribution

It introduces and compares three levels of LLM-generated feedback against a baseline, revealing the impact of feedback structure on student performance.

Findings

LLM-generated feedback leads to faster solutions than no feedback.

Less guided feedback slightly outperforms more guided feedback.

Feedback structure significantly influences student problem-solving behavior.

Abstract

When programming students encounter errors in their code, compiler messages or static analysis output often provide limited guidance, particularly for novice programmers. Personalized feedback from instructors can be effective but does not scale well. Recent advances in large language models (LLMs) enable automated feedback generation at scale. This study examines whether LLM-generated feedback with different levels of guidance is associated with differences in students' problem-solving behavior. We analyze effects on time to solution and number of attempts, and examine whether these effects differ by programming experience. We design three feedback types and compare them to a baseline in which students receive only compiler error messages. Results from an online programming course show that LLM-generated feedback is associated with faster time to solution compared to the no-feedback baseline, with less guided feedback showing slightly stronger effects. Overall, the findings suggest that feedback structure plays an important role in how students progress toward correct solutions and motivate further work on adaptive feedback designs and longer-term learning outcomes.