Measuring Hong Kong Massive Multi-Task Language Understanding

TL;DR

This paper introduces HKMMLU, a comprehensive benchmark for evaluating multilingual and socio-cultural understanding of LLMs in Hong Kong's unique linguistic context, revealing significant performance gaps and influencing future model development.

Contribution

The paper presents HKMMLU, a novel multi-task benchmark tailored for Hong Kong's linguistic and cultural landscape, including 66 subjects and translation tasks, to evaluate LLMs' capabilities.

Findings

DeepSeek-V3 achieves up to 75% accuracy, still below MMLU and CMMLU.

Model performance is affected by question language, size, and prompting strategies.

Significant performance gap indicates need for improved Hong Kong-specific language models.

Abstract

Multilingual understanding is crucial for the cross-cultural applicability of Large Language Models (LLMs). However, evaluation benchmarks designed for Hong Kong's unique linguistic landscape, which combines Traditional Chinese script with Cantonese as the spoken form and its cultural context, remain underdeveloped. To address this gap, we introduce HKMMLU, a multi-task language understanding benchmark that evaluates Hong Kong's linguistic competence and socio-cultural knowledge. The HKMMLU includes 26,698 multi-choice questions across 66 subjects, organized into four categories: Science, Technology, Engineering, and Mathematics (STEM), Social Sciences, Humanities, and Other. To evaluate the multilingual understanding ability of LLMs, 90,550 Mandarin-Cantonese translation tasks were additionally included. We conduct comprehensive experiments on GPT-4o, Claude 3.7 Sonnet, and 18 open-source LLMs of varying sizes on HKMMLU. The results show that the best-performing model, DeepSeek-V3, struggles to achieve an accuracy of 75\%, significantly lower than that of MMLU and CMMLU. This performance gap highlights the need to improve LLMs' capabilities in Hong Kong-specific language and knowledge domains. Furthermore, we investigate how question language, model size, prompting strategies, and question and reasoning token lengths affect model performance. We anticipate that HKMMLU will significantly advance the development of LLMs in multilingual and cross-cultural contexts, thereby enabling broader and more impactful applications.