Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming

TL;DR

This study investigates how AI code generators like Codex influence novice learners' programming performance and retention, finding they improve coding success without harming manual skills and may enhance long-term learning.

Contribution

The paper provides empirical evidence on the effects of AI code generators on novice programming learning, highlighting performance improvements and potential retention benefits.

Findings

Codex increased code completion rates and scores.

No negative impact on manual code-modification performance.

Prior access to Codex benefited learners with higher Scratch scores on retention tests.

Abstract

AI code generators like OpenAI Codex have the potential to assist novice programmers by generating code from natural language descriptions, however, over-reliance might negatively impact learning and retention. To explore the implications that AI code generators have on introductory programming, we conducted a controlled experiment with 69 novices (ages 10-17). Learners worked on 45 Python code-authoring tasks, for which half of the learners had access to Codex, each followed by a code-modification task. Our results show that using Codex significantly increased code-authoring performance (1.15x increased completion rate and 1.8x higher scores) while not decreasing performance on manual code-modification tasks. Additionally, learners with access to Codex during the training phase performed slightly better on the evaluation post-tests conducted one week later, although this difference did not reach statistical significance. Of interest, learners with higher Scratch pre-test scores performed significantly better on retention post-tests, if they had prior access to Codex.