Experiences from Integrating Large Language Model Chatbots into the Classroom

TL;DR

This study examines how students interact with unfiltered GPT-4 chatbots integrated into online courses, revealing limited engagement outside LLM-focused classes and highlighting the need for tailored LLM experiences in education.

Contribution

It provides empirical insights into student usage patterns of unfiltered LLM chatbots in diverse educational settings and discusses implications for designing effective LLM integrations.

Findings

High usage in LLM-focused courses

Limited engagement in non-LLM courses

Most usage driven by a few superusers

Abstract

In the present study, we provided students an unfiltered access to a state-of-the-art large language model (LLM) chatbot. The chatbot was intentionally designed to mimic proprietary commercial chatbots such as ChatGPT where the chatbot has not been tailored for the educational context; the underlying engine was OpenAI GPT-4. The chatbot was integrated into online learning materials of three courses. One of the courses focused on software engineering with LLMs, while the two other courses were not directly related to LLMs. Our results suggest that only a minority of students engage with the chatbot in the courses that do not relate to LLMs. At the same time, unsurprisingly, nearly all students in the LLM-focused course leveraged the chatbot. In all courses, the majority of the LLM usage came from a few superusers, whereas the majority of the students did not heavily use the chatbot even though it was readily available and effectively provided a free access to the OpenAI GPT-4 model. We also observe that in addition to students using the chatbot for course-specific purposes, many use the chatbot for their own purposes. These results suggest that the worst fears of educators -- all students overrelying on LLMs -- did not materialize even when the chatbot access was unfiltered. We finally discuss potential reasons for the low usage, suggesting the need for more tailored and scaffolded LLM experiences targeted for specific types of student use cases.