The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers

TL;DR

This study investigates how generative AI tools impact novice programmers' problem-solving, revealing benefits for some students and potential harms for others, especially those with existing metacognitive difficulties.

Contribution

It extends prior research by analyzing the effects of GenAI on novice programming behavior through observational and eye-tracking methods, highlighting disparities among students.

Findings

Students who used GenAI effectively completed problems and ignored unhelpful suggestions.

Struggling students faced compounded metacognitive difficulties and experienced an illusion of competence.

GenAI can both aid and hinder novice programmers depending on their metacognitive skills.

Abstract

Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who accelerated and students who struggled. Students who accelerated were able to use GenAI to create code they already intended to make and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.