Chatbot Conversations in Physics Education: Using Artificial Intelligence to Analyze Student Reasoning through Computational Grounded Theory

TL;DR

This paper demonstrates how Computational Grounded Theory can analyze chatbot-student interactions to uncover misconceptions and reasoning patterns in physics education, leveraging NLP and machine learning techniques.

Contribution

It introduces a novel CGT pipeline combining NLP, clustering, and supervised learning to analyze large-scale physics student dialogues from an AI chatbot.

Findings

Identified persistent misconceptions in relativity and quantum physics

Revealed patterns in student question phrasing and uncertainty expression

Showed CGT's potential for scalable analysis of educational dialogues

Abstract

This study applies Computational Grounded Theory (CGT) to analyze student misconceptions using interaction data from an AI-powered chatbot deployed in a university-level Modern Physics course. The chatbot - the UTA Study Buddy Bot - engaged students in peer-like problem-solving conversations throughout the semester, generating a rich dataset of over 10 million tokens. To explore patterns in student reasoning and identify recurring conceptual difficulties, we implemented a CGT pipeline that combined natural language processing, unsupervised clustering of sentence-level vector embeddings, human interpretation of emergent themes, and supervised learning to evaluate the generalizability of identified categories. Preliminary results revealed persistent misconceptions in areas such as relativistic momentum and quantum energy levels, along with distinctive trends in how students phrased their questions and expressed uncertainty. These findings underscore the potential of CGT as a scalable, theory-aligned approach for extracting insights from chatbot dialogues and guiding the development of more adaptive, AI-driven educational tools in physics instruction.