SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark

TL;DR

SuperCLUE is a comprehensive Chinese benchmark for large language models that evaluates performance across diverse real-world tasks and user preferences, highlighting the limitations of accuracy metrics alone.

Contribution

This paper introduces SuperCLUE, the first Chinese benchmark integrating user queries, open-ended, and closed-ended questions to better assess LLMs in practical scenarios.

Findings

Accuracy on closed-ended questions alone is insufficient to reflect human preferences.

Open-ended and closed-ended questions together better predict user preferences.

GPT-4 reliably evaluates human preferences in Chinese LLM tasks.

Abstract

Large language models (LLMs) have shown the potential to be integrated into human daily lives. Therefore, user preference is the most critical criterion for assessing LLMs' performance in real-world scenarios. However, existing benchmarks mainly focus on measuring models' accuracy using multi-choice questions, which limits the understanding of their capabilities in real applications. We fill this gap by proposing a comprehensive Chinese benchmark SuperCLUE, named after another popular Chinese LLM benchmark CLUE. SuperCLUE encompasses three sub-tasks: actual users' queries and ratings derived from an LLM battle platform (CArena), open-ended questions with single and multiple-turn dialogues (OPEN), and closed-ended questions with the same stems as open-ended single-turn ones (CLOSE). Our study shows that accuracy on closed-ended questions is insufficient to reflect human preferences achieved on open-ended ones. At the same time, they can complement each other to predict actual user preferences. We also demonstrate that GPT-4 is a reliable judge to automatically evaluate human preferences on open-ended questions in a Chinese context. Our benchmark will be released at https://www.CLUEbenchmarks.com