Leveraging LLMs to Assess Tutor Moves in Real-Life Dialogues: A Feasibility Study

TL;DR

This study explores the use of advanced language models to automatically identify and evaluate key tutoring actions in real-world math tutoring dialogues, demonstrating high accuracy and practical feasibility.

Contribution

It introduces a scalable, AI-based approach for assessing tutor moves in authentic settings, with a novel prompting strategy and reproducible LLM prompts.

Findings

Models reliably detected praise and math errors with over 82% accuracy.

AI assessments closely matched human judgments, with 73-89% agreement.

Proposes cost-effective prompting methods for real-world tutoring analysis.

Abstract

Tutoring improves student achievement, but identifying and studying what tutoring actions are most associated with student learning at scale based on audio transcriptions is an open research problem. This present study investigates the feasibility and scalability of using generative AI to identify and evaluate specific tutor moves in real-life math tutoring. We analyze 50 randomly selected transcripts of college-student remote tutors assisting middle school students in mathematics. Using GPT-4, GPT-4o, GPT-4-turbo, Gemini-1.5-pro, and LearnLM, we assess tutors' application of two tutor skills: delivering effective praise and responding to student math errors. All models reliably detected relevant situations, for example, tutors providing praise to students (94-98% accuracy) and a student making a math error (82-88% accuracy) and effectively evaluated the tutors' adherence to tutoring best practices, aligning closely with human judgments (83-89% and 73-77%, respectively). We propose a cost-effective prompting strategy and discuss practical implications for using large language models to support scalable assessment in authentic settings. This work further contributes LLM prompts to support reproducibility and research in AI-supported learning.