Exploring LLMs for Predicting Tutor Strategy and Student Outcomes in Dialogues

TL;DR

This paper investigates the capability of large language models like Llama 3 and GPT-4o to predict tutor strategies and student outcomes in math tutoring dialogues, revealing current limitations and the importance of tutor behavior for student success.

Contribution

It is the first study to evaluate modern LLMs for predicting tutor strategies and student outcomes in dialogue-based tutoring, highlighting the need for improved methods.

Findings

LLMs struggle to accurately predict future tutor strategies

Tutor strategies are highly indicative of student outcomes

Current models require enhancement for effective prediction

Abstract

Tutoring dialogues have gained significant attention in recent years, given the prominence of online learning and the emerging tutoring abilities of artificial intelligence (AI) agents powered by large language models (LLMs). Recent studies have shown that the strategies used by tutors can have significant effects on student outcomes, necessitating methods to predict how tutors will behave and how their actions impact students. However, few works have studied predicting tutor strategy in dialogues. Therefore, in this work we investigate the ability of modern LLMs, particularly Llama 3 and GPT-4o, to predict both future tutor moves and student outcomes in dialogues, using two math tutoring dialogue datasets. We find that even state-of-the-art LLMs struggle to predict future tutor strategy while tutor strategy is highly indicative of student outcomes, outlining a need for more powerful methods to approach this task.