Not All Students Engage Alike: Multi-Institution Patterns in GenAI Tutor Use

TL;DR

This study analyzes diverse student engagement patterns with GenAI Tutors across multiple institutions, revealing significant heterogeneity influenced by institutional and disciplinary factors, and highlights the importance of understanding engagement for responsible deployment.

Contribution

It introduces a framework for analyzing student engagement with GenAI Tutors and uncovers how engagement varies across contexts and over time, informing scalable and responsible implementation.

Findings

10.4% of sessions showed shallow engagement with copy-pasting

Students from highly selective institutions engaged more deeply

Students' engagement types evolved over time and varied by context

Abstract

The emergence of generative artificial intelligence (GenAI) has created unprecedented opportunities to provide individualized learning support in classrooms as automated tutoring systems at scale. However, concerns have been raised that students may engage with these tools in ways that do not support learning. Moreover, student engagement with GenAI Tutors may vary across instructional contexts, potentially leading to unequal learning experiences. In this study, we utilize de-identified student interaction logs from an existing GenAI Tutor and the learning management system in which it is embedded. We systematically examined student engagement (N = 11,406) with the tool across 200 classes in ten post-secondary institutions through a two-stage pipeline: First, we identified four distinct engagement types at the conversation session level. In particular, 10.4% of them were "shallow engagement" where copy-pasting behavior was prevalent. Then, at the student level, we show that students transitioned across engagement types over time. However, students who exhibited shallow engagement with the tool were more likely to remain in this mode, whereas those who engaged deeply with the tool transitioned more flexibly across engagement types. Finally, at both the session and student levels, we show substantial heterogeneity in student engagement across institution selectivity and course disciplines. In particular, students from highly selective institutions were more likely to exhibit deep engagement. Together, our study advances the understanding of how GenAI Tutors are used in authentic educational settings and provides a framework for analyzing student engagement with GenAI Tutors, with implications for responsible implementation at scale.