Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?
Andreas Scholl, Natalie Kiesler

TL;DR
This study examines how students interact with the SCRIPT Chatbot in programming education, revealing patterns in feedback requests and how well the chatbot's responses meet student needs, informing future AI learning tools.
Contribution
It provides empirical insights into student-chatbot interactions and feedback preferences in programming education using a novel ChatGPT-4-based system.
Findings
Students follow a specific sequence in feedback requests.
Chatbot responses match requested feedback types 75% of the time.
The system adheres to prompt constraints, balancing guidance and flexibility.
Abstract
Building on prior research on Generative AI (GenAI) and related tools for programming education, we developed SCRIPT, a chatbot based on ChatGPT-4o-mini, to support novice learners. SCRIPT allows for open-ended interactions and structured guidance through predefined prompts. We evaluated the tool via an experiment with 136 students from an introductory programming course at a large German university and analyzed how students interacted with SCRIPT while solving programming tasks with a focus on their feedback preferences. The results reveal that students' feedback requests seem to follow a specific sequence. Moreover, the chatbot responses aligned well with students' requested feedback types (in 75%), and it adhered to the system prompt constraints. These insights inform the design of GenAI-based learning support systems and highlight challenges in balancing guidance and flexibility in…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAI in Service Interactions · Online Learning and Analytics
