Changes in Coding Behavior and Performance Since the Introduction of LLMs
Yufan Zhang, Jaromir Savelka, Seth Copen Goldstein, and Michael Conway

TL;DR
This study examines how student coding behaviors and performance have changed over five years, especially after the introduction of LLMs like ChatGPT, indicating decreased learning and productivity possibly due to over-reliance on these tools.
Contribution
It provides the first longitudinal analysis of student coding behavior before and after LLMs, highlighting significant behavioral shifts and their potential impact on learning outcomes.
Findings
Increased final submission length post-2022
Decreased score improvements between submissions
Correlations between behavioral changes and performance
Abstract
The widespread availability of large language models (LLMs) has changed how students engage with coding and problem-solving. While these tools may increase student productivity, they also make it more difficult for instructors to assess students' learning and effort. In this quasi-longitudinal study, we analyze five years of student source code submissions in a graduate-level cloud computing course, focusing on an assignment that remained unchanged and examining students' behavior during the period spanning five semesters before the release of ChatGPT and five semesters after. Student coding behavior has changed significantly since Fall 2022. The length of their final submissions increased. Between consecutive submissions, average edit distances increased while average score improvement decreased, suggesting that both student productivity and learning have decreased after ChatGPT's…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsArtificial Intelligence in Healthcare and Education · Online Learning and Analytics · Intelligent Tutoring Systems and Adaptive Learning
