Advancing the Evaluation of Traditional Chinese Language Models: Towards a Comprehensive Benchmark Suite

TL;DR

This paper introduces a comprehensive benchmark suite for evaluating Traditional Chinese language models, covering diverse tasks and enabling fair comparison of models like GPT-3.5, Taiwan-LLaMa, and proprietary models.

Contribution

It presents a new set of benchmarks tailored for Traditional Chinese, leveraging existing datasets and covering multiple tasks, which was previously lacking in the field.

Findings

Model 7-C performs comparably to GPT-3.5 on several tasks.

The benchmark suite is open-sourced for community use.

Evaluation across diverse tasks highlights strengths and weaknesses of models.

Abstract

The evaluation of large language models is an essential task in the field of language understanding and generation. As language models continue to advance, the need for effective benchmarks to assess their performance has become imperative. In the context of Traditional Chinese, there is a scarcity of comprehensive and diverse benchmarks to evaluate the capabilities of language models, despite the existence of certain benchmarks such as DRCD, TTQA, CMDQA, and FGC dataset. To address this gap, we propose a novel set of benchmarks that leverage existing English datasets and are tailored to evaluate language models in Traditional Chinese. These benchmarks encompass a wide range of tasks, including contextual question-answering, summarization, classification, and table understanding. The proposed benchmarks offer a comprehensive evaluation framework, enabling the assessment of language models' capabilities across different tasks. In this paper, we evaluate the performance of GPT-3.5, Taiwan-LLaMa-v1.0, and Model 7-C, our proprietary model, on these benchmarks. The evaluation results highlight that our model, Model 7-C, achieves performance comparable to GPT-3.5 with respect to a part of the evaluated capabilities. In an effort to advance the evaluation of language models in Traditional Chinese and stimulate further research in this field, we have open-sourced our benchmark and opened the model for trial.