Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT

TL;DR

This study analyzes how 213 students at a German university interact with ChatGPT-3.5 while solving programming exercises, revealing diverse chat patterns that can inform teaching practices in introductory programming courses.

Contribution

It provides a detailed analysis of student-chat interactions with LLMs during programming tasks, highlighting patterns and implications for education.

Findings

Diverse interaction patterns observed in student chats

Both supportive and concerning chat behaviors identified

Insights to improve teaching strategies for programming courses

Abstract

Large Language Models (LLMs) have taken the world by storm, and students are assumed to use related tools at a great scale. In this research paper we aim to gain an understanding of how introductory programming students chat with LLMs and related tools, e.g., ChatGPT-3.5. To address this goal, computing students at a large German university were motivated to solve programming exercises with the assistance of ChatGPT as part of their weekly introductory course exercises. Then students (n=213) submitted their chat protocols (with 2335 prompts in sum) as data basis for this analysis. The data was analyzed w.r.t. the prompts, frequencies, the chats' progress, contents, and other use pattern, which revealed a great variety of interactions, both potentially supportive and concerning. Learning about students' interactions with ChatGPT will help inform and align teaching practices and instructions for future introductory programming courses in higher education.