How Do Students Interact with an LLM-powered Virtual Teaching Assistant in Different Educational Settings?

TL;DR

This study examines how students interact with Jill, an LLM-powered virtual teaching assistant, across various educational settings, revealing insights into question complexity, usage patterns, and course-specific behaviors.

Contribution

It provides a detailed analysis of student interactions with an LLM-based TA across multiple courses, highlighting factors influencing question types and engagement levels.

Findings

Jill supports a wide range of cognitive questions, encouraging higher-order thinking.

Usage frequency varies significantly across different course deployments.

Question types are influenced by specific course contexts and student needs.

Abstract

Jill Watson, a virtual teaching assistant powered by LLMs, answers student questions and engages them in extended conversations on courseware provided by the instructors. In this paper, we analyze student interactions with Jill across multiple courses and colleges, focusing on the types and complexity of student questions based on Bloom's Revised Taxonomy and tool usage patterns. We find that, by supporting a wide range of cognitive demands, Jill encourages students to engage in sophisticated, higher-order cognitive questions. However, the frequency of usage varies significantly across deployments, and the types of questions asked depend on course-specific contexts. These findings pave the way for future work on AI-driven educational tools tailored to individual learning styles and course structure, potentially enhancing both the teaching and learning experience in classrooms.