From Solver to Tutor: Evaluating the Pedagogical Intelligence of LLMs with KMP-Bench

TL;DR

This paper introduces KMP-Bench, a comprehensive benchmark for evaluating the pedagogical abilities of large language models in K-8 mathematics tutoring, highlighting current models' strengths and weaknesses.

Contribution

The paper presents KMP-Bench, a novel multi-faceted benchmark with datasets and evaluation modules to assess LLMs' pedagogical skills in math tutoring.

Findings

Leading LLMs excel at verifiable tasks but struggle with pedagogical principles.

Fine-tuning on KMP-Pile improves models' pedagogical performance.

KMP-Bench reveals gaps in current LLMs' teaching capabilities.

Abstract

Large Language Models (LLMs) show significant potential in AI mathematical tutoring, yet current evaluations often rely on simplistic metrics or narrow pedagogical scenarios, failing to assess comprehensive, multi-turn teaching effectiveness. In this paper, we introduce KMP-Bench, a comprehensive K-8 Mathematical Pedagogical Benchmark designed to assess LLMs from two complementary perspectives. The first module, KMP-Dialogue, evaluates holistic pedagogical capabilities against six core principles (e.g., Challenge, Explanation, Feedback), leveraging a novel multi-turn dialogue dataset constructed by weaving together diverse pedagogical components. The second module, KMP-Skills, provides a granular assessment of foundational tutoring abilities, including multi-turn problem-solving, error detection and correction, and problem generation. Our evaluations on KMP-Bench reveal a key disparity: while leading LLMs excel at tasks with verifiable solutions, they struggle with the nuanced application of pedagogical principles. Additionally, we present KMP-Pile, a large-scale (150K) dialogue dataset. Models fine-tuned on KMP-Pile show substantial improvement on KMP-Bench, underscoring the value of pedagogically-rich training data for developing more effective AI math tutors.