\textsc{CantoNLU}: A benchmark for Cantonese natural language understanding

TL;DR

This paper introduces CantoNLU, a comprehensive benchmark for Cantonese NLP tasks, providing datasets, baseline models, and analysis to advance research in this under-resourced language.

Contribution

It presents the first multi-task Cantonese NLU benchmark, along with baseline models and analysis, to support future NLP research in Cantonese.

Findings

Cantonese-adapted models outperform other models overall.

Monolingual Cantonese models excel at syntactic tasks.

Mandarin models remain competitive with limited Cantonese data.

Abstract

Cantonese, although spoken by millions, remains under-resourced due to policy and diglossia. To address this scarcity of evaluation frameworks for Cantonese, we introduce \textsc{\textbf{CantoNLU}}, a benchmark for Cantonese natural language understanding (NLU). This novel benchmark spans seven tasks covering syntax and semantics, including word sense disambiguation, linguistic acceptability judgment, language detection, natural language inference, sentiment analysis, part-of-speech tagging, and dependency parsing. In addition to the benchmark, we provide model baseline performance across a set of models: a Mandarin model without Cantonese training, two Cantonese-adapted models obtained by continual pre-training a Mandarin model on Cantonese text, and a monolingual Cantonese model trained from scratch. Results show that Cantonese-adapted models perform best overall, while monolingual models perform better on syntactic tasks. Mandarin models remain competitive in certain settings, indicating that direct transfer may be sufficient when Cantonese domain data is scarce. We release all datasets, code, and model weights to facilitate future research in Cantonese NLP.