LLMs Struggle with NLI for Perfect Aspect: A Cross-Linguistic Study in Chinese and Japanese

TL;DR

This study investigates the challenges faced by large language models in understanding the perfect aspect in Chinese and Japanese, revealing significant limitations in temporal inference across languages.

Contribution

It introduces a new cross-linguistic NLI dataset focused on perfect aspect in Chinese and Japanese, highlighting the need for multilingual temporal semantics evaluation.

Findings

LLMs struggle with subtle tense and reference-time shifts

Models show limited understanding of perfect aspect in non-English languages

Cross-linguistic differences impact LLM performance in temporal inference

Abstract

Unlike English, which uses distinct forms (e.g., had, has, will have) to mark the perfect aspect across tenses, Chinese and Japanese lack separate grammatical forms for tense within the perfect aspect, which complicates Natural Language Inference (NLI). Focusing on the perfect aspect in these languages, we construct a linguistically motivated, template-based NLI dataset (1,350 pairs per language). Experiments reveal that even advanced LLMs struggle with temporal inference, particularly in detecting subtle tense and reference-time shifts. These findings highlight model limitations and underscore the need for cross-linguistic evaluation in temporal semantics. Our dataset is available at https://github.com/Lujie2001/CrossNLI.