SwissGov-RSD: A Human-annotated, Cross-lingual Benchmark for Token-level Recognition of Semantic Differences Between Related Documents

TL;DR

This paper introduces SwissGov-RSD, a novel cross-lingual benchmark dataset for token-level recognition of semantic differences between related documents, highlighting the performance gap of current models in this task.

Contribution

It provides the first naturalistic, multi-parallel, cross-lingual dataset with human annotations for semantic difference recognition at the token level.

Findings

Current models perform poorly on the new benchmark compared to monolingual and synthetic tasks.

The dataset reveals significant challenges in cross-lingual semantic difference detection.

Models show a large performance gap, indicating room for improvement.

Abstract

Recognizing semantic differences across documents is crucial for text generation evaluation and content alignment, especially in cross-lingual settings. However, as a standalone task, it has received little attention. We address this by introducing SwissGov-RSD, the first naturalistic, document-level, cross-lingual dataset for semantic difference recognition. It encompasses a total of 224 multi-parallel documents in English--German, English--French, and English--Italian with token-level difference annotations by human annotators. We evaluate a variety of open-source and closed-source large language models as well as encoder models across different fine-tuning settings on this new benchmark. Our results show that current automatic approaches perform poorly compared to their performance on monolingual, sentence-level, and synthetic benchmarks, revealing a considerable gap for both LLMs and encoder models. We make our code and dataset publicly available.