Cracking the Code: Evaluating Zero-Shot Prompting Methods for Providing Programming Feedback

TL;DR

This paper evaluates various zero-shot prompt engineering techniques for improving programming feedback quality from large language models, focusing on error detection in R programming through systematic prompt variation.

Contribution

It introduces a framework for assessing zero-shot prompts and identifies effective strategies for enhancing feedback accuracy in LLMs.

Findings

Stepwise prompts increase feedback precision.

Omitting explicit data analysis instructions improves error detection.

Prompt design significantly impacts feedback quality.

Abstract

Despite the growing use of large language models (LLMs) for providing feedback, limited research has explored how to achieve high-quality feedback. This case study introduces an evaluation framework to assess different zero-shot prompt engineering methods. We varied the prompts systematically and analyzed the provided feedback on programming errors in R. The results suggest that prompts suggesting a stepwise procedure increase the precision, while omitting explicit specifications about which provided data to analyze improves error identification.