MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors

TL;DR

MathTutorBench is an open-source benchmark designed to evaluate the pedagogical abilities of AI tutoring models across various dimensions, highlighting the trade-offs between subject expertise and teaching quality.

Contribution

It introduces a comprehensive, easy-to-use evaluation framework for AI tutors, including datasets, metrics, and a reward model for assessing pedagogical quality.

Findings

Subject expertise does not guarantee good teaching.

Longer dialogues challenge tutoring effectiveness.

Specialization influences pedagogical performance.

Abstract

Evaluating the pedagogical capabilities of AI-based tutoring models is critical for making guided progress in the field. Yet, we lack a reliable, easy-to-use, and simple-to-run evaluation that reflects the pedagogical abilities of models. To fill this gap, we present MathTutorBench, an open-source benchmark for holistic tutoring model evaluation. MathTutorBench contains a collection of datasets and metrics that broadly cover tutor abilities as defined by learning sciences research in dialog-based teaching. To score the pedagogical quality of open-ended teacher responses, we train a reward model and show it can discriminate expert from novice teacher responses with high accuracy. We evaluate a wide set of closed- and open-weight models on MathTutorBench and find that subject expertise, indicated by solving ability, does not immediately translate to good teaching. Rather, pedagogy and subject expertise appear to form a trade-off that is navigated by the degree of tutoring specialization of the model. Furthermore, tutoring appears to become more challenging in longer dialogs, where simpler questioning strategies begin to fail. We release the benchmark, code, and leaderboard openly to enable rapid benchmarking of future models.