Do Tutors Learn from Equity Training and Can Generative AI Assess It?

TL;DR

This study evaluates how generative AI, specifically large language models, can assess and improve tutors' equity-related skills in online education, showing promising results in performance and confidence gains.

Contribution

It introduces a novel application of LLMs for assessing tutor responses to equity scenarios and provides a dataset for future research in scalable equity training assessment.

Findings

Marginally significant increase in tutor confidence post-training

GPT-4o with few-shot learning effectively assesses tutor performance

Dataset and prompts made publicly available for further research

Abstract

Equity is a core concern of learning analytics. However, applications that teach and assess equity skills, particularly at scale are lacking, often due to barriers in evaluating language. Advances in generative AI via large language models (LLMs) are being used in a wide range of applications, with this present work assessing its use in the equity domain. We evaluate tutor performance within an online lesson on enhancing tutors' skills when responding to students in potentially inequitable situations. We apply a mixed-method approach to analyze the performance of 81 undergraduate remote tutors. We find marginally significant learning gains with increases in tutors' self-reported confidence in their knowledge in responding to middle school students experiencing possible inequities from pretest to posttest. Both GPT-4o and GPT-4-turbo demonstrate proficiency in assessing tutors ability to predict and explain the best approach. Balancing performance, efficiency, and cost, we determine that few-shot learning using GPT-4o is the preferred model. This work makes available a dataset of lesson log data, tutor responses, rubrics for human annotation, and generative AI prompts. Future work involves leveling the difficulty among scenarios and enhancing LLM prompts for large-scale grading and assessment.