Understanding Translationese in Cross-Lingual Summarization

TL;DR

This paper investigates how translationese affects cross-lingual summarization, revealing its impact on evaluation, model performance, and dataset construction, and offers guidelines for future research in the field.

Contribution

It systematically analyzes the influence of translationese on CLS datasets, evaluation, and models, providing insights and recommendations for future dataset and model development.

Findings

Translationese in test sets causes evaluation discrepancies.

Training on translationese can harm real-world model performance.

Machine-translated documents are useful for low-resource CLS systems.

Abstract

Given a document in a source language, cross-lingual summarization (CLS) aims at generating a concise summary in a different target language. Unlike monolingual summarization (MS), naturally occurring source-language documents paired with target-language summaries are rare. To collect large-scale CLS data, existing datasets typically involve translation in their creation. However, the translated text is distinguished from the text originally written in that language, i.e., translationese. In this paper, we first confirm that different approaches of constructing CLS datasets will lead to different degrees of translationese. Then we systematically investigate how translationese affects CLS model evaluation and performance when it appears in source documents or target summaries. In detail, we find that (1) the translationese in documents or summaries of test sets might lead to the discrepancy between human judgment and automatic evaluation; (2) the translationese in training sets would harm model performance in real-world applications; (3) though machine-translated documents involve translationese, they are very useful for building CLS systems on low-resource languages under specific training strategies. Lastly, we give suggestions for future CLS research including dataset and model developments. We hope that our work could let researchers notice the phenomenon of translationese in CLS and take it into account in the future.