Characterizing Students' LLM Usage Behaviors and Their Association with Learning in Critical Thinking Tasks

TL;DR

This study analyzes how students use large language models in a research course and investigates the impact of different usage patterns on their learning outcomes across multiple assessments.

Contribution

It introduces a refined categorization of student LLM usage types and examines their association with learning performance in a real educational setting.

Findings

Different LLM usage types correlate with varying levels of student initiative.

Frequency and type of LLM use are linked to performance on midterm assessments.

Refined usage categories provide deeper insights into student engagement with LLMs.

Abstract

Large language models (LLMs) are becoming increasingly embedded in students' learning practices, yet much of what is known about how students use LLMs and how this usage impacts learning comes from problem-solving domains or constrained experimental settings. We present an analysis of data on LLM usage collected during two offerings of a research-oriented course where students learn to read, reason about, and critique academic papers. Without restrictions on whether or how to use LLMs, students reported their LLM usage practices when asked to do these activities as a series of homework assignments during the course. This paper extends prior work done on data from a single offering of the same course by presenting a refined bottom-up categorization of LLM usage types, cross-labeled by the extent of student initiative these usages entail. Furthermore, we examine how LLM use impacts student learning, measured by performance on three midterms, looking at factors such as frequency and type of usage.