From Self-Crafted to Engineered Prompts: Student Evaluations of AI-Generated Feedback in Introductory Physics

TL;DR

This study compares student preferences for AI-generated feedback in physics education, finding structured prompts with prompt engineering and effective feedback are most favored, highlighting the importance of prompt design in educational AI tools.

Contribution

It introduces a comparative analysis of different prompt types for AI feedback in physics education, emphasizing the role of prompt engineering and feedback principles.

Findings

Structured prompts are preferred over self-crafted prompts.

Feedback with prompt engineering and effective principles is most favored.

Students show strong preferences, either liking or disliking the feedback.

Abstract

The abilities of Generative-Artificial Intelligence (AI) to produce real-time, sophisticated responses across diverse contexts has promised a huge potential in physics education, particularly in providing customized feedback. In this study, we investigate around 1200 introductory students' preferences about AI-feedback generated from three distinct prompt types: (a) self-crafted, (b) entailing foundational prompt-engineering techniques, and (c) entailing foundational prompt-engineering techniques along with principles of effective-feedback. The results highlight an overwhelming fraction of students preferring feedback generated using structured prompts, with those entailing combined features of prompt engineering and effective feedback to be favored most. However, the popular choice also elicited stronger preferences with students either liking or disliking the feedback. Students also ranked the feedback generated using their self-crafted prompts as the least preferred choice. Students' second preferences given their first choice and implications of the results such as the need to incorporate prompt engineering in introductory courses are discussed.