Analyzing Undergraduate Problem-Solving in Physics Through Interaction With an AI Chatbot

TL;DR

This study explores how an AI chatbot with Socratic questioning impacts physics problem-solving skills and confidence among undergraduates, showing increased question specificity and positive perceptions of learning effectiveness.

Contribution

Introduces a Socratic AI chatbot for physics education that enhances reasoning and provides detailed analytics, demonstrating scalability for instruction and research.

Findings

Question specificity increased from 10-15% to 100% during interactions.

Question specificity correlated positively with expected course grade (r=0.43).

Students rated the chatbot as effective for knowledge and overall learning.

Abstract

Providing individualized scaffolding for physics problem solving at scale remains an instructional challenge. We investigate (1) students' perceptions of a Socratic Artificial Intelligence (AI) chatbot's impact on problem-solving skills and confidence and (2) how the specificity of students' questions during tutoring relates to performance. We deployed a custom Socratic AI chatbot in a large-enrollment introductory mechanics course at a Midwestern public university, logging full dialogue transcripts from 150 first-year STEM majors. Post-interaction surveys revealed median ratings of 4.0/5 for knowledge-based skills and 3.4/5 for overall effectiveness. Transcript analysis showed question specificity rose from approximately 10-15% in the first turn to 100% by the final turn, and specificity correlated positively with self reported expected course grade (Pearson r = 0.43). These findings demonstrate that AI-driven Socratic dialogue not only fosters expert-like reasoning but also generates fine-grained analytics for physics education research, establishing a scalable dual-purpose tool for instruction and learning analytics.