EduEVAL-DB: A Role-Based Dataset for Pedagogical Risk Evaluation in Educational Explanations

TL;DR

EduEVAL-DB is a novel dataset featuring teacher role-based explanations and risk annotations, designed to improve automatic pedagogical evaluation and AI tutoring systems across K-12 subjects.

Contribution

This paper introduces EduEVAL-DB, a comprehensive dataset with expert-annotated pedagogical explanations and risk labels, supporting development of pedagogical risk detection models.

Findings

Benchmarking shows Gemini 2.5 Pro outperforms Llama 3.1 8B in pedagogical risk detection.

Supervised fine-tuning on EduEVAL-DB enhances model performance on risk assessment.

The dataset effectively supports evaluation of AI explanations in educational contexts.

Abstract

This work introduces EduEVAL-DB, a dataset based on teacher roles designed to support the evaluation and training of automatic pedagogical evaluators and AI tutors for instructional explanations. The dataset comprises 854 explanations corresponding to 139 questions from a curated subset of the ScienceQA benchmark, spanning science, language, and social science across K-12 grade levels. For each question, one human-teacher explanation is provided and six are generated by LLM-simulated teacher roles. These roles are inspired by instructional styles and shortcomings observed in real educational practice and are instantiated via prompt engineering. We further propose a pedagogical risk rubric aligned with established educational standards, operationalizing five complementary risk dimensions: factual correctness, explanatory depth and completeness, focus and relevance, student-level appropriateness, and ideological bias. All explanations are annotated with binary risk labels through a semi-automatic process with expert teacher review. Finally, we present preliminary validation experiments to assess the suitability of EduEVAL-DB for evaluation. We benchmark a state-of-the-art education-oriented model (Gemini 2.5 Pro) against a lightweight local Llama 3.1 8B model and examine whether supervised fine-tuning on EduEVAL-DB supports pedagogical risk detection using models deployable on consumer hardware.