Unpacking Graduate Students' Learning Experience with Generative AI Teaching Assistant in A Quantitative Methodology Course

TL;DR

This study explores graduate students' interactions with a generative AI teaching assistant in a quantitative methods course, revealing usage patterns, question types, and differences based on students' backgrounds to inform better AI integration in education.

Contribution

It provides empirical insights into how graduate students engage with AI teaching assistants, highlighting usage trends and cognitive levels of questions across diverse student backgrounds.

Findings

Students showed higher AI usage at course start and end, with less during middle weeks.

Most questions focused on knowledge and comprehension, fewer on deeper thinking.

Students with weaker math backgrounds used AI more but with less structured inquiries.

Abstract

The study was conducted in an Advanced Quantitative Research Methods course involving 20 graduate students. During the course, student inquiries made to the AI were recorded and coded using Bloom's taxonomy and the CLEAR framework. A series of independent sample t-tests and poisson regression analyses were employed to analyse the characteristics of different questions asked by students with different backgrounds. Post course interviews were conducted with 10 students to gain deeper insights into their perceptions. The findings revealed a U-shaped pattern in students' use of the AI assistant, with higher usage at the beginning and towards the end of the course, and a decrease in usage during the middle weeks. Most questions posed to the AI focused on knowledge and comprehension levels, with fewer questions involving deeper cognitive thinking. Students with a weaker mathematical foundation used the AI assistant more frequently, though their inquiries tended to lack explicit and logical structure compared to those with a strong mathematical foundation, who engaged less with the tool. These patterns suggest the need for targeted guidance to optimise the effectiveness of AI tools for students with varying levels of academic proficiency.