ZhuJiu: A Multi-dimensional, Multi-faceted Chinese Benchmark for Large Language Models

TL;DR

ZhuJiu is a comprehensive Chinese benchmark for evaluating large language models across multiple abilities, using diverse methods, and addressing data leakage, to provide accurate and systematic assessment results.

Contribution

We introduce ZhuJiu, a multi-dimensional, multi-faceted Chinese benchmark for LLM evaluation, covering 7 ability dimensions and 51 tasks, with new focus on knowledge ability.

Findings

Evaluated 10 mainstream LLMs with ZhuJiu benchmark.

Demonstrated the effectiveness of multi-method evaluation approach.

Provided publicly accessible benchmark and leaderboard.

Abstract

The unprecedented performance of large language models (LLMs) requires comprehensive and accurate evaluation. We argue that for LLMs evaluation, benchmarks need to be comprehensive and systematic. To this end, we propose the ZhuJiu benchmark, which has the following strengths: (1) Multi-dimensional ability coverage: We comprehensively evaluate LLMs across 7 ability dimensions covering 51 tasks. Especially, we also propose a new benchmark that focuses on knowledge ability of LLMs. (2) Multi-faceted evaluation methods collaboration: We use 3 different yet complementary evaluation methods to comprehensively evaluate LLMs, which can ensure the authority and accuracy of the evaluation results. (3) Comprehensive Chinese benchmark: ZhuJiu is the pioneering benchmark that fully assesses LLMs in Chinese, while also providing equally robust evaluation abilities in English. (4) Avoiding potential data leakage: To avoid data leakage, we construct evaluation data specifically for 37 tasks. We evaluate 10 current mainstream LLMs and conduct an in-depth discussion and analysis of their results. The ZhuJiu benchmark and open-participation leaderboard are publicly released at http://www.zhujiu-benchmark.com/ and we also provide a demo video at https://youtu.be/qypkJ89L1Ic.