CLEVA: Chinese Language Models EVAluation Platform

TL;DR

CLEVA is a comprehensive, user-friendly platform designed to evaluate Chinese language models across multiple dimensions, addressing challenges like standardization, contamination, and comparability in current Chinese LLM assessments.

Contribution

CLEVA introduces a standardized evaluation workflow, a regularly updated leaderboard, and contamination mitigation strategies for Chinese LLMs, enhancing fairness and comprehensiveness.

Findings

Validated with 23 Chinese LLMs demonstrating effectiveness.

Provides a standardized, easy-to-use evaluation process.

Ensures data freshness and reduces contamination risks.

Abstract

With the continuous emergence of Chinese Large Language Models (LLMs), how to evaluate a model's capabilities has become an increasingly significant issue. The absence of a comprehensive Chinese benchmark that thoroughly assesses a model's performance, the unstandardized and incomparable prompting procedure, and the prevalent risk of contamination pose major challenges in the current evaluation of Chinese LLMs. We present CLEVA, a user-friendly platform crafted to holistically evaluate Chinese LLMs. Our platform employs a standardized workflow to assess LLMs' performance across various dimensions, regularly updating a competitive leaderboard. To alleviate contamination, CLEVA curates a significant proportion of new data and develops a sampling strategy that guarantees a unique subset for each leaderboard round. Empowered by an easy-to-use interface that requires just a few mouse clicks and a model API, users can conduct a thorough evaluation with minimal coding. Large-scale experiments featuring 23 Chinese LLMs have validated CLEVA's efficacy.