Backretrieval: An Image-Pivoted Evaluation Metric for Cross-Lingual Text Representations Without Parallel Corpora

TL;DR

Backretrieval introduces an image-based evaluation metric for cross-lingual text representations, enabling assessment without parallel corpora, and demonstrates high correlation with ground truth metrics through experiments and a case study.

Contribution

The paper proposes a novel image-pivoted metric for evaluating cross-lingual embeddings without requiring parallel data, validated by experiments and human study.

Findings

Backretrieval correlates highly with ground truth metrics.

Statistically significant improvements over baseline methods.

Effective evaluation demonstrated on a recipe dataset without parallel data.

Abstract

Cross-lingual text representations have gained popularity lately and act as the backbone of many tasks such as unsupervised machine translation and cross-lingual information retrieval, to name a few. However, evaluation of such representations is difficult in the domains beyond standard benchmarks due to the necessity of obtaining domain-specific parallel language data across different pairs of languages. In this paper, we propose an automatic metric for evaluating the quality of cross-lingual textual representations using images as a proxy in a paired image-text evaluation dataset. Experimentally, Backretrieval is shown to highly correlate with ground truth metrics on annotated datasets, and our analysis shows statistically significant improvements over baselines. Our experiments conclude with a case study on a recipe dataset without parallel cross-lingual data. We illustrate how to judge cross-lingual embedding quality with Backretrieval, and validate the outcome with a small human study.