A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese

TL;DR

This paper introduces ZhoBLiMP, a comprehensive Chinese linguistic minimal pair benchmark, and evaluates various Chinese language models using a new length-normalized metric, revealing challenges in modeling complex linguistic phenomena.

Contribution

It presents the largest Chinese minimal pair benchmark and proposes SLLN-LP, a novel metric to reduce length bias in evaluating language models.

Findings

Chinese LMs struggle with anaphor, quantifiers, and ellipsis even at 32B parameters.

SLLN-LP effectively mitigates length bias in evaluation.

Evaluation metrics should consider linguistic relations and sentence length biases.

Abstract

We present ZhoBLiMP, the largest linguistic minimal pair benchmark for Chinese, with over 100 paradigms, ranging from topicalization to the \textit{Ba} construction. We then train from scratch a suite of Chinese language models (LMs) with different tokenizers, parameter sizes, and token volumes, to study the learning curves of LMs on Chinese. To mitigate the biases introduced by unequal lengths of the sentences in a minimal pair, we propose a new metric named sub-linear length normalized log-probabilities (SLLN-LP). Using SLLN-LP as the metric, our results show that \textsc{Anaphor}, \textsc{Quantifiers}, and \textsc{Ellipsis} in Chinese are difficult for LMs even up to 32B parameters, and that SLLN-LP successfully mitigates biases in ZhoBLiMP, JBLiMP and BLiMP. We conclude that future evaluations should be more carefully designed to consider the intricate relations between linking functions, LMs, and targeted minimal pairs.