Polish Natural Language Inference and Factivity -- an Expert-based Dataset and Benchmarks

TL;DR

This paper introduces a new Polish NLI dataset focused on factivity, evaluates transformer models and linguistic features, and highlights challenges in complex cases like entitlement and non-factive verbs.

Contribution

It provides the first expert-annotated Polish factivity NLI dataset and benchmarks the performance of BERT-based models and linguistic features on this task.

Findings

BERT-based models achieved around 89% F1 score.

Linguistic feature-based models achieved around 91% F1 score.

Complex cases like entitlement and non-factive verbs remain challenging.

Abstract

Despite recent breakthroughs in Machine Learning for Natural Language Processing, the Natural Language Inference (NLI) problems still constitute a challenge. To this purpose we contribute a new dataset that focuses exclusively on the factivity phenomenon; however, our task remains the same as other NLI tasks, i.e. prediction of entailment, contradiction or neutral (ECN). The dataset contains entirely natural language utterances in Polish and gathers 2,432 verb-complement pairs and 309 unique verbs. The dataset is based on the National Corpus of Polish (NKJP) and is a representative sample in regards to frequency of main verbs and other linguistic features (e.g. occurrence of internal negation). We found that transformer BERT-based models working on sentences obtained relatively good results ($\approx89\%$ F1 score). Even though better results were achieved using linguistic features ($\approx91\%$ F1 score), this model requires more human labour (humans in the loop) because features were prepared manually by expert linguists. BERT-based models consuming only the input sentences show that they capture most of the complexity of NLI/factivity. Complex cases in the phenomenon - e.g. cases with entitlement (E) and non-factive verbs - remain an open issue for further research.