From BERT to LLMs: Comparing and Understanding Chinese Classifier Prediction in Language Models

TL;DR

This paper evaluates whether large language models understand Chinese classifiers by testing their prediction abilities and analyzing the impact of sentence elements, revealing that LLMs underperform BERT even after fine-tuning.

Contribution

It systematically compares LLMs and BERT in Chinese classifier prediction and investigates the effects of sentence context and fine-tuning on model performance.

Findings

LLMs perform worse than BERT in Chinese classifier prediction

Prediction accuracy improves with information about the following noun

Bidirectional attention mechanisms like BERT's confer advantages in this task

Abstract

Classifiers are an important and defining feature of the Chinese language, and their correct prediction is key to numerous educational applications. Yet, whether the most popular Large Language Models (LLMs) possess proper knowledge the Chinese classifiers is an issue that has largely remain unexplored in the Natural Language Processing (NLP) literature. To address such a question, we employ various masking strategies to evaluate the LLMs' intrinsic ability, the contribution of different sentence elements, and the working of the attention mechanisms during prediction. Besides, we explore fine-tuning for LLMs to enhance the classifier performance. Our findings reveal that LLMs perform worse than BERT, even with fine-tuning. The prediction, as expected, greatly benefits from the information about the following noun, which also explains the advantage of models with a bidirectional attention mechanism such as BERT.