How Good is BLI as an Alignment Measure: A Study in Word Embedding Paradigm

TL;DR

This study critically examines the effectiveness of Bilingual Lexicon Induction (BLI) as a measure for evaluating the alignment of word embedding spaces, proposing new methods and analyzing various models across resource levels.

Contribution

It introduces a stem-based BLI approach and a vocabulary pruning technique, providing a nuanced evaluation of embedding alignment methods in multilingual contexts.

Findings

BLI does not always accurately measure true alignment.

Combined alignment techniques often outperform individual methods.

Multilingual embeddings can outperform aligned monolingual models in low-resource scenarios.

Abstract

Sans a dwindling number of monolingual embedding studies originating predominantly from the low-resource domains, it is evident that multilingual embedding has become the de facto choice due to its adaptability to the usage of code-mixed languages, granting the ability to process multilingual documents in a language-agnostic manner, as well as removing the difficult task of aligning monolingual embeddings. But is this victory complete? Are the multilingual models better than aligned monolingual models in every aspect? Can the higher computational cost of multilingual models always be justified? Or is there a compromise between the two extremes? Bilingual Lexicon Induction is one of the most widely used metrics in terms of evaluating the degree of alignment between two embedding spaces. In this study, we explore the strengths and limitations of BLI as a measure to evaluate the degree of alignment of two embedding spaces. Further, we evaluate how well traditional embedding alignment techniques, novel multilingual models, and combined alignment techniques perform BLI tasks in the contexts of both high-resource and low-resource languages. In addition to that, we investigate the impact of the language families to which the pairs of languages belong. We identify that BLI does not measure the true degree of alignment in some cases and we propose solutions for them. We propose a novel stem-based BLI approach to evaluate two aligned embedding spaces that take into account the inflected nature of languages as opposed to the prevalent word-based BLI techniques. Further, we introduce a vocabulary pruning technique that is more informative in showing the degree of the alignment, especially performing BLI on multilingual embedding models. Often, combined embedding alignment techniques perform better while in certain cases multilingual embeddings perform better (mainly low-resource language cases).