Proposition from the Perspective of Chinese Language: A Chinese Proposition Classification Evaluation Benchmark

TL;DR

This paper introduces a comprehensive Chinese proposition classification benchmark, PEACE, and evaluates various models, revealing the importance of semantic modeling and highlighting current limitations in cross-domain transferability and model performance.

Contribution

It proposes a new multi-level classification system for Chinese propositions, creates the PEACE dataset, and evaluates multiple models, including ChatGPT, on this benchmark.

Findings

BERT shows good classification ability but limited transferability.

ChatGPT performs poorly but can improve with more proposition information.

Semantic features are crucial for accurate proposition classification.

Abstract

Existing propositions often rely on logical constants for classification. Compared with Western languages that lean towards hypotaxis such as English, Chinese often relies on semantic or logical understanding rather than logical connectives in daily expressions, exhibiting the characteristics of parataxis. However, existing research has rarely paid attention to this issue. And accurately classifying these propositions is crucial for natural language understanding and reasoning. In this paper, we put forward the concepts of explicit and implicit propositions and propose a comprehensive multi-level proposition classification system based on linguistics and logic. Correspondingly, we create a large-scale Chinese proposition dataset PEACE from multiple domains, covering all categories related to propositions. To evaluate the Chinese proposition classification ability of existing models and explore their limitations, We conduct evaluations on PEACE using several different methods including the Rule-based method, SVM, BERT, RoBERTA, and ChatGPT. Results show the importance of properly modeling the semantic features of propositions. BERT has relatively good proposition classification capability, but lacks cross-domain transferability. ChatGPT performs poorly, but its classification ability can be improved by providing more proposition information. Many issues are still far from being resolved and require further study.