Changes in Coding Behavior and Performance Since the Introduction of LLMs

TL;DR

This study examines how student coding behaviors and performance have changed over five years, especially after the introduction of LLMs like ChatGPT, indicating decreased learning and productivity possibly due to over-reliance on these tools.

Contribution

It provides the first longitudinal analysis of student coding behavior before and after LLMs, highlighting significant behavioral shifts and their potential impact on learning outcomes.

Findings

Increased final submission length post-2022

Decreased score improvements between submissions

Correlations between behavioral changes and performance

Abstract

The widespread availability of large language models (LLMs) has changed how students engage with coding and problem-solving. While these tools may increase student productivity, they also make it more difficult for instructors to assess students' learning and effort. In this quasi-longitudinal study, we analyze five years of student source code submissions in a graduate-level cloud computing course, focusing on an assignment that remained unchanged and examining students' behavior during the period spanning five semesters before the release of ChatGPT and five semesters after. Student coding behavior has changed significantly since Fall 2022. The length of their final submissions increased. Between consecutive submissions, average edit distances increased while average score improvement decreased, suggesting that both student productivity and learning have decreased after ChatGPT's release. Additionally, there are statistically significant correlations between these behavioral changes and their overall performance. Although we cannot definitively attribute them to LLM misuse, they are consistent with our hypothesis that some students are over-reliant on LLMs, which is negatively affecting their learning outcomes. Our findings raise an alarm around the first generation of graduates in the age of LLMs, calling upon both educators and employers to reflect on their evaluation methods for genuine expertise and productivity.