Does Summary Evaluation Survive Translation to Other Languages?
Spencer Braun, Oleg Vasilyev, Neslihan Iskender, John Bohannon

TL;DR
This paper investigates whether summary evaluation datasets created in one language can be reliably used in other languages through machine translation, finding limited transferability for most evaluation methods.
Contribution
The study evaluates the effectiveness of translating a summarization dataset into multiple languages and assesses the statistical equivalence of evaluation methods across translations.
Findings
Limited transferability of evaluation methods across languages
Some potential for dataset reuse in similar languages
Most evaluation methods are not statistically equivalent across translations
Abstract
The creation of a quality summarization dataset is an expensive, time-consuming effort, requiring the production and evaluation of summaries by both trained humans and machines. If such effort is made in one language, it would be beneficial to be able to use it in other languages without repeating human annotations. To investigate how much we can trust machine translation of such a dataset, we translate the English dataset SummEval to seven languages and compare performance across automatic evaluation measures. We explore equivalence testing as the appropriate statistical paradigm for evaluating correlations between human and automated scoring of summaries. While we find some potential for dataset reuse in languages similar to the source, most summary evaluation methods are not found to be statistically equivalent across translations.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Data Mining Algorithms and Applications
