Understanding Student Interaction with AI-Powered Next-Step Hints: Strategies and Challenges

TL;DR

This paper explores how students interact with AI-powered next-step hints in programming education, revealing strategies for managing unhelpful hints and providing insights to improve hint design.

Contribution

It offers a detailed analysis of student interactions with AI hints, identifies common strategies for handling unhelpful hints, and provides a publicly available dataset for future research.

Findings

Identified 16 common interaction scenarios

Students adapt hints by modifying code or hints

Provided insights for improving hint design

Abstract

Automated feedback generation plays a crucial role in enhancing personalized learning experiences in computer science education. Among different types of feedback, next-step hint feedback is particularly important, as it provides students with actionable steps to progress towards solving programming tasks. This study investigates how students interact with an AI-driven next-step hint system in an in-IDE learning environment. We gathered and analyzed a dataset from 34 students solving Kotlin tasks, containing detailed hint interaction logs. We applied process mining techniques and identified 16 common interaction scenarios. Semi-structured interviews with 6 students revealed strategies for managing unhelpful hints, such as adapting partial hints or modifying code to generate variations of the same hint. These findings, combined with our publicly available dataset, offer valuable opportunities for future research and provide key insights into student behavior, helping improve hint design for enhanced learning support.