Hint-Writing with Deferred AI Assistance: Fostering Critical Engagement in Data Science Education

TL;DR

This study explores how deferred AI assistance in hint-writing activities enhances students' debugging skills and critical engagement in data science education.

Contribution

It introduces and evaluates a deferred AI support design that improves hint quality and student learning outcomes compared to other methods.

Findings

Deferred AI assistance yields the highest-quality hints.

Students better identify mistakes with deferred AI support.

Activities promote critical engagement and debugging skills.

Abstract

Generating hints for incorrect code is a cognitively demanding task that fosters learning and metacognitive development. This study investigates three designs for personalized, scalable, and reflective hint-writing activities within a data science course: (i) writing a hint independently, (ii) writing a hint with on-demand AI assistance, and (iii) deferred AI assistance, in which students first write a hint independently and then revise it with the help of an AI-generated one. We examine how AI support can scaffold the learning process without diminishing students' productive cognitive effort. Through a randomized controlled experiment with graduate-level students (N=97), we found that deferring AI assistance leads to the highest-quality hints. Further, this design helps students identify a wide range of mistakes they otherwise struggle to identify without any AI assistance. Students valued these activities as opportunities to practice debugging and critically engage with AI outputs--skills that are now critical for learners to acquire as programming becomes increasingly automated and the use of AI for learning grows. Our findings also highlight key considerations for designing student-AI collaborative learning experiences to sustain student engagement, maintain appropriate cognitive load, and mitigate negative effects of AI, such as introducing redundancies and extraneous information into student work.