Beyond the Hype: A Cautionary Tale of ChatGPT in the Programming Classroom

TL;DR

This paper examines the use of ChatGPT in university programming education, analyzing its impact on student learning, assessment, and academic integrity, and offers guidance for educators to adapt teaching strategies.

Contribution

It provides a novel analysis of ChatGPT's practical use in programming courses, highlighting challenges and strategies for fostering critical skills and responsible AI use.

Findings

ChatGPT often produces common programming errors.

It influences assessment methods and student engagement.

Guidelines for creating challenging exercises and promoting integrity.

Abstract

Due to the proliferation of Large Language Models research and the use of various Artificial Intelligence (AI) tools, the field of information systems (IS) and computer science (CS) has evolved. The use of tools such as ChatGPT to complete various student programming exercises (e.g., in Python) and assignments has gained prominence amongst various academic institutions. However, recent literature has suggested that the use of ChatGPT in academia is problematic and the impact on teaching and learning should be further scrutinized. More specifically, little is known about how ChatGPT can be practically used with code (programming) writing to complete programming exercises amongst IS and CS undergraduate university students. Furthermore, the paper provides insights for academics who teach programming to create more challenging exercises and how to engage responsibly in the use of ChatGPT to promote classroom integrity. In this paper, we used Complex Adaptive Systems (CAS) theory as a theoretical guide to understand the various dynamics through classroom code demonstrations. Using ChatGPT 3.5, we analyzed the various practical programming examples from past IS exercises and compared those with memos created by tutors and lecturers in a university setting. This paper highlights common ways of assessment, programming errors created by ChatGPT and the potential consideration for IS academics to ensure the development of critical programming skills among students.