Edu-Values: Towards Evaluating the Chinese Education Values of Large Language Models

TL;DR

Edu-Values is a comprehensive Chinese education values benchmark for evaluating large language models, revealing cultural influences, challenges in ethics, and improving alignment through external knowledge integration.

Contribution

This paper introduces Edu-Values, the first Chinese education values benchmark with diverse question types, and demonstrates its effectiveness in evaluating and enhancing LLMs.

Findings

Chinese LLMs outperform English LLMs due to cultural differences

LLMs struggle with teachers' ethics and philosophy

Using Edu-Values for external knowledge improves LLM alignment

Abstract

In this paper, we present Edu-Values, the first Chinese education values evaluation benchmark that includes seven core values: professional philosophy, teachers' professional ethics, education laws and regulations, cultural literacy, educational knowledge and skills, basic competencies and subject knowledge. We meticulously design 1,418 questions, covering multiple-choice, multi-modal question answering, subjective analysis, adversarial prompts, and Chinese traditional culture (short answer) questions. We conduct human feedback based automatic evaluation over 21 state-of-the-art (SoTA) LLMs, and highlight three main findings: (1) due to differences in educational culture, Chinese LLMs outperform English LLMs, with Qwen 2 ranking the first with a score of 81.37; (2) LLMs often struggle with teachers' professional ethics and professional philosophy; (3) leveraging Edu-Values to build an external knowledge repository for RAG significantly improves LLMs' alignment. This demonstrates the effectiveness of the proposed benchmark.