Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learning

TL;DR

This study explores and refines seven cognitive engagement techniques to improve deep learning in novice programmers interacting with AI-generated code, emphasizing the most effective method of guided, step-by-step problem-solving.

Contribution

The paper introduces a systematic design process for engagement techniques and identifies the most effective method for fostering deeper learning with AI-generated code.

Findings

Guided step-by-step problem-solving is most effective.

Techniques increase learners' ability to apply concepts independently.

Iterative refinement improves engagement and learning outcomes.

Abstract

Novice programmers are increasingly relying on Large Language Models (LLMs) to generate code for learning programming concepts. However, this interaction can lead to superficial engagement, giving learners an illusion of learning and hindering skill development. To address this issue, we conducted a systematic design exploration to develop seven cognitive engagement techniques aimed at promoting deeper engagement with AI-generated code. In this paper, we describe our design process, the initial seven techniques and results from a between-subjects study (N=82). We then iteratively refined the top techniques and further evaluated them through a within-subjects study (N=42). We evaluate the friction each technique introduces, their effectiveness in helping learners apply concepts to isomorphic tasks without AI assistance, and their success in aligning learners' perceived and actual coding abilities. Ultimately, our results highlight the most effective technique: guiding learners through the step-by-step problem-solving process, where they engage in an interactive dialog with the AI, prompting what needs to be done at each stage before the corresponding code is revealed.