Measuring Changes in Instructor Class Design and Student Learning After the Release of Large Language Models (LLMs)

TL;DR

This study investigates how the release of Large Language Models (LLMs) has altered instructor class design and student learning, using mixed methods including surveys and grade data analysis.

Contribution

It provides empirical insights into the impact of LLMs on higher education practices and student achievement, informing policy and pedagogical strategies.

Findings

Documented shifts in student and faculty perceptions of LLM use.

Triangulated grade data shows changes in learning achievement post-LLM release.

Identified patterns in class design adaptations in response to LLMs.

Abstract

Student use of Generative AI (GenAI) products in completing their classwork, with or without their professors' knowledge and/or approval, has resulted in substantial shifts in higher education. While GenAI use is widespread, its impact on student study methods, faculty course development, grade reporting, and overall learning is not well documented. This is a mixed-methods, multi-course study using retrospective quantitative analysis, instructor surveys, and anonymous student surveys at a university in the New England region of the United States. This research seeks to identify and document patterns in student and faculty perceptions of, and experiences in, the use of LLMs as a learning tool inside and outside of the university classroom. Alongside quantitative and thematic analysis of both faculty and student survey responses, historical grade data as reported to the university registrar is used to triangulate the phenomenon of learning achievement in pre- and post-LLM eras. It is hoped that this research can serve as a pilot study for a broader set of institutions. Results from this study can inform GenAI policy for professors, universities, and other educational institutions that are trying to maximize student learning in the age of AI.