The Limitations of Cross-language Word Embeddings Evaluation

TL;DR

This paper investigates the limitations of current cross-language word embeddings evaluation methods, revealing poor correlation between intrinsic and extrinsic assessments and questioning the validity of human references as ground truth.

Contribution

It constructs English-Russian datasets for evaluation and demonstrates the lack of correlation among different benchmarks, highlighting evaluation challenges.

Findings

Scores on different intrinsic benchmarks do not correlate.

Human references may not be reliable ground truth.

Evaluation methods need reconsideration for cross-language embeddings.

Abstract

The aim of this work is to explore the possible limitations of existing methods of cross-language word embeddings evaluation, addressing the lack of correlation between intrinsic and extrinsic cross-language evaluation methods. To prove this hypothesis, we construct English-Russian datasets for extrinsic and intrinsic evaluation tasks and compare performances of 5 different cross-language models on them. The results say that the scores even on different intrinsic benchmarks do not correlate to each other. We can conclude that the use of human references as ground truth for cross-language word embeddings is not proper unless one does not understand how do native speakers process semantics in their cognition.