A "watch your replay videos" reflection assignment on comparing programming without versus with generative AI: learning about programming, critical AI use and limitations, and reflection

TL;DR

This study used a novel video reflection assignment to help students understand and critically evaluate their programming processes with and without generative AI, fostering metacognitive skills and lifelong learning.

Contribution

It introduces a comparative video reflection method that enhances understanding of programming, AI limitations, and reflection skills, extending beyond traditional AI use instruction.

Findings

Students gained insights into debugging and help-seeking behaviors.

Students recognized AI limitations and learned critical AI use strategies.

Reflection fostered future goal setting and lifelong learning habits.

Abstract

Generative AI is disrupting computing education. Most interventions focus on teaching GenAI use rather than helping students understand how AI changes their programming process. We designed and deployed a novel comparative video reflection assignment adapting the Describe, Examine, then Articulate Learning (DEAL) framework. In an introductory software engineering course, students recorded themselves programming during their team project two times: first without, then with using generative AI. Students then analyzed their own videos using a scaffolded set of reflection questions, including on their programming process and human, internet, and AI help-seeking. We conducted a qualitative thematic analysis of the reflections, finding students developed insights about planning, debugging, and help-seeking behaviors that transcended AI use. Students reported learning to slow down and understand before writing or generating code, recognized patterns in their problem-solving approaches, and articulated specific process improvements. Students also learned and reflected on AI limits and downsides, and strategies to use AI more critically, including better prompting but also to benefit their learning instead of just completing tasks. Unexpectedly, the comparative reflection also scaffolded reflection on programming not involving AI use, and even led to students spontaneously setting future goals to adopt video and other regular reflection. This work demonstrates structured reflection on programming session videos can develop metacognitive skills essential for programming with and without generative AI and also lifelong learning in our evolving field.