EduEval: A Hierarchical Cognitive Benchmark for Evaluating Large Language Models in Chinese Education

TL;DR

EduEval is a comprehensive hierarchical benchmark designed to evaluate large language models in Chinese K-12 education, covering cognitive skills, authenticity of tasks, and scale, to ensure safe and effective deployment.

Contribution

It introduces the EduAbility Taxonomy, integrates authentic educational data, and provides a large-scale evaluation framework for LLMs in Chinese education.

Findings

Models excel at factual tasks but struggle with dialogue and creativity.

Open source models outperform proprietary ones on complex reasoning.

Few-shot prompting effectiveness varies across cognitive dimensions.

Abstract

Large language models (LLMs) demonstrate significant potential for educational applications. However, their unscrutinized deployment poses risks to educational standards, underscoring the need for rigorous evaluation. We introduce EduEval, a comprehensive hierarchical benchmark for evaluating LLMs in Chinese K-12 education. This benchmark makes three key contributions: (1) Cognitive Framework: We propose the EduAbility Taxonomy, which unifies Bloom's Taxonomy and Webb's Depth of Knowledge to organize tasks across six cognitive dimensions including Memorization, Understanding, Application, Reasoning, Creativity, and Ethics. (2) Authenticity: Our benchmark integrates real exam questions, classroom conversation, student essays, and expert-designed prompts to reflect genuine educational challenges; (3) Scale: EduEval comprises 24 distinct task types with over 11,000 questions spanning primary to high school levels. We evaluate 14 leading LLMs under both zero-shot and few-shot settings, revealing that while models perform well on factual tasks, they struggle with classroom dialogue classification and exhibit inconsistent results in creative content generation. Interestingly, several open source models outperform proprietary systems on complex educational reasoning. Few-shot prompting shows varying effectiveness across cognitive dimensions, suggesting that different educational objectives require tailored approaches. These findings provide targeted benchmarking metrics for developing LLMs specifically optimized for diverse Chinese educational tasks.