TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Students

TL;DR

TeachTune enables teachers to review and improve pedagogical agents by simulating diverse student profiles with LLMs, reducing workload and increasing coverage of student types.

Contribution

This paper introduces TeachTune, a novel pipeline for simulating diverse student profiles to review pedagogical agents, addressing limitations of existing review methods.

Findings

Simulated students' behaviors closely match input knowledge and motivation levels.

Teachers using TeachTune experienced lower task load.

TeachTune increased student profile coverage compared to baseline.

Abstract

Large language models (LLMs) can empower teachers to build pedagogical conversational agents (PCAs) customized for their students. As students have different prior knowledge and motivation levels, teachers must review the adaptivity of their PCAs to diverse students. Existing chatbot reviewing methods (e.g., direct chat and benchmarks) are either manually intensive for multiple iterations or limited to testing only single-turn interactions. We present TeachTune, where teachers can create simulated students and review PCAs by observing automated chats between PCAs and simulated students. Our technical pipeline instructs an LLM-based student to simulate prescribed knowledge levels and traits, helping teachers explore diverse conversation patterns. Our pipeline could produce simulated students whose behaviors correlate highly to their input knowledge and motivation levels within 5% and 10% accuracy gaps. Thirty science teachers designed PCAs in a between-subjects study, and using TeachTune resulted in a lower task load and higher student profile coverage over a baseline.