Generating Feedback-Ladders for Logical Errors in Programming using Large Language Models

TL;DR

This paper introduces a method using large language models to generate layered feedback for programming errors, allowing tailored and progressive guidance that enhances student learning and addresses limitations of existing feedback approaches.

Contribution

The paper proposes a novel layered feedback generation approach using LLMs, considering student context and enabling progressive feedback levels for programming assignments.

Findings

Layered feedback improves student learning outcomes.

Higher-level feedback has diminishing effectiveness for advanced submissions.

Teachers can select appropriate feedback levels based on student context.

Abstract

In feedback generation for logical errors in programming assignments, large language model (LLM)-based methods have shown great promise. These methods ask the LLM to generate feedback given the problem statement and a student's (buggy) submission. There are several issues with these types of methods. First, the generated feedback messages are often too direct in revealing the error in the submission and thus diminish valuable opportunities for the student to learn. Second, they do not consider the student's learning context, i.e., their previous submissions, current knowledge, etc. Third, they are not layered since existing methods use a single, shared prompt for all student submissions. In this paper, we explore using LLMs to generate a "feedback-ladder", i.e., multiple levels of feedback for the same problem-submission pair. We evaluate the quality of the generated feedback-ladder via a user study with students, educators, and researchers. We have observed diminishing effectiveness for higher-level feedback and higher-scoring submissions overall in the study. In practice, our method enables teachers to select an appropriate level of feedback to show to a student based on their personal learning context, or in a progressive manner to go more detailed if a higher-level feedback fails to correct the student's error.