QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluation

TL;DR

QualBench is a comprehensive Chinese domain-specific benchmark using qualification exam questions to evaluate LLMs, revealing current performance gaps and highlighting the importance of localized knowledge and targeted improvements.

Contribution

This paper introduces QualBench, the first multi-domain Chinese QA benchmark based on qualification exams, providing a new standardized evaluation framework for Chinese LLMs.

Findings

Chinese LLMs outperform non-Chinese models in domain-specific tasks.

Average accuracy of 53.98% indicates significant room for improvement.

Prompt engineering and fine-tuning enhance model performance.

Abstract

The rapid advancement of Chinese LLMs underscores the need for vertical-domain evaluations to ensure reliable applications. However, existing benchmarks often lack domain coverage and provide limited insights into the Chinese working context. Leveraging qualification exams as a unified framework for expertise evaluation, we introduce QualBench, the first multi-domain Chinese QA benchmark dedicated to localized assessment of Chinese LLMs. The dataset includes over 17,000 questions across six vertical domains, drawn from 24 Chinese qualifications to align with national policies and professional standards. Results reveal an interesting pattern of Chinese LLMs consistently surpassing non-Chinese models, with the Qwen2.5 model outperforming the more advanced GPT-4o, emphasizing the value of localized domain knowledge in meeting qualification requirements. The average accuracy of 53.98% reveals the current gaps in domain coverage within model capabilities. Furthermore, we identify performance degradation caused by LLM crowdsourcing, assess data contamination, and illustrate the effectiveness of prompt engineering and model fine-tuning, suggesting opportunities for future improvements through multi-domain RAG and Federated Learning.