Meta-evaluation of comparability metrics using parallel corpora

TL;DR

This paper introduces a meta-evaluation method for comparability metrics of parallel corpora, assessing their reliability by correlating monolingual scores across source and target texts to ensure consistent domain distance measurements.

Contribution

It proposes a novel meta-evaluation approach that assesses the reliability of comparability metrics through correlation analysis on parallel corpora, enabling better metric selection and optimization.

Findings

The method yields consistent results across different datasets.

Correlations indicate the reliability of comparability metrics.

The approach helps optimize metric parameters for better corpus evaluation.

Abstract

Metrics for measuring the comparability of corpora or texts need to be developed and evaluated systematically. Applications based on a corpus, such as training Statistical MT systems in specialised narrow domains, require finding a reasonable balance between the size of the corpus and its consistency, with controlled and benchmarked levels of comparability for any newly added sections. In this article we propose a method that can meta-evaluate comparability metrics by calculating monolingual comparability scores separately on the 'source' and 'target' sides of parallel corpora. The range of scores on the source side is then correlated (using Pearson's r coefficient) with the range of 'target' scores; the higher the correlation - the more reliable is the metric. The intuition is that a good metric should yield the same distance between different domains in different languages. Our method gives consistent results for the same metrics on different data sets, which indicates that it is reliable and can be used for metric comparison or for optimising settings of parametrised metrics.