Compositional Evaluation on Japanese Textual Entailment and Similarity

TL;DR

This paper introduces JSICK, a Japanese NLI/STS dataset, and a stress-test for compositional inference, revealing that current pre-trained models are insensitive to word order and case particles in Japanese.

Contribution

It provides the first Japanese NLI/STS dataset and a novel stress-test to evaluate models' sensitivity to syntactic variations in Japanese.

Findings

Models show insensitivity to word order and case particles.

Multilingual models perform variably across languages.

Japanese-specific dataset enables better linguistic evaluation.

Abstract

Natural Language Inference (NLI) and Semantic Textual Similarity (STS) are widely used benchmark tasks for compositional evaluation of pre-trained language models. Despite growing interest in linguistic universals, most NLI/STS studies have focused almost exclusively on English. In particular, there are no available multilingual NLI/STS datasets in Japanese, which is typologically different from English and can shed light on the currently controversial behavior of language models in matters such as sensitivity to word order and case particles. Against this background, we introduce JSICK, a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. We also present a stress-test dataset for compositional inference, created by transforming syntactic structures of sentences in JSICK to investigate whether language models are sensitive to word order and case particles. We conduct baseline experiments on different pre-trained language models and compare the performance of multilingual models when applied to Japanese and other languages. The results of the stress-test experiments suggest that the current pre-trained language models are insensitive to word order and case marking.