Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?

TL;DR

This study examines how students interact with the SCRIPT Chatbot in programming education, revealing patterns in feedback requests and how well the chatbot's responses meet student needs, informing future AI learning tools.

Contribution

It provides empirical insights into student-chatbot interactions and feedback preferences in programming education using a novel ChatGPT-4-based system.

Findings

Students follow a specific sequence in feedback requests.

Chatbot responses match requested feedback types 75% of the time.

The system adheres to prompt constraints, balancing guidance and flexibility.

Abstract

Building on prior research on Generative AI (GenAI) and related tools for programming education, we developed SCRIPT, a chatbot based on ChatGPT-4o-mini, to support novice learners. SCRIPT allows for open-ended interactions and structured guidance through predefined prompts. We evaluated the tool via an experiment with 136 students from an introductory programming course at a large German university and analyzed how students interacted with SCRIPT while solving programming tasks with a focus on their feedback preferences. The results reveal that students' feedback requests seem to follow a specific sequence. Moreover, the chatbot responses aligned well with students' requested feedback types (in 75%), and it adhered to the system prompt constraints. These insights inform the design of GenAI-based learning support systems and highlight challenges in balancing guidance and flexibility in AI-assisted tools.