FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Models

TL;DR

FoundaBench is a comprehensive benchmark for evaluating Chinese large language models' fundamental knowledge across diverse subjects, revealing performance disparities and guiding future improvements.

Contribution

This paper introduces FoundaBench, a new benchmark with 3354 questions to assess Chinese LLMs' fundamental knowledge, employing novel evaluation protocols.

Findings

Models trained on Chinese data outperform others.

Significant gap between reasoning and memory recall abilities.

FoundaBench sets a new standard for LLM knowledge evaluation.

Abstract

In the burgeoning field of large language models (LLMs), the assessment of fundamental knowledge remains a critical challenge, particularly for models tailored to Chinese language and culture. This paper introduces FoundaBench, a pioneering benchmark designed to rigorously evaluate the fundamental knowledge capabilities of Chinese LLMs. FoundaBench encompasses a diverse array of 3354 multiple-choice questions across common sense and K-12 educational subjects, meticulously curated to reflect the breadth and depth of everyday and academic knowledge. We present an extensive evaluation of 12 state-of-the-art LLMs using FoundaBench, employing both traditional assessment methods and our CircularEval protocol to mitigate potential biases in model responses. Our results highlight the superior performance of models pre-trained on Chinese corpora, and reveal a significant disparity between models' reasoning and memory recall capabilities. The insights gleaned from FoundaBench evaluations set a new standard for understanding the fundamental knowledge of LLMs, providing a robust framework for future advancements in the field.