X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs

TL;DR

This paper introduces X-PARADE, a novel cross-lingual dataset for paragraph-level information divergence and entailment, and evaluates various methods including LLMs, highlighting their current limitations compared to humans.

Contribution

The paper presents the first cross-lingual dataset for paragraph-level information divergence and analyzes multiple approaches, including LLM prompting, for this complex task.

Findings

Methods vary in handling inferable information

All evaluated methods underperform compared to humans

Aligned Wikipedia paragraphs reflect real-world divergences

Abstract

Understanding when two pieces of text convey the same information is a goal touching many subproblems in NLP, including textual entailment and fact-checking. This problem becomes more complex when those two pieces of text are in different languages. Here, we introduce X-PARADE (Cross-lingual Paragraph-level Analysis of Divergences and Entailments), the first cross-lingual dataset of paragraph-level information divergences. Annotators label a paragraph in a target language at the span level and evaluate it with respect to a corresponding paragraph in a source language, indicating whether a given piece of information is the same, new, or new but can be inferred. This last notion establishes a link with cross-language NLI. Aligned paragraphs are sourced from Wikipedia pages in different languages, reflecting real information divergences observed in the wild. Armed with our dataset, we investigate a diverse set of approaches for this problem, including token alignment from machine translation, textual entailment methods that localize their decisions, and prompting LLMs. Our results show that these methods vary in their capability to handle inferable information, but they all fall short of human performance.