Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective

TL;DR

This paper introduces a local, domain-specific approach to cross-lingual lexical alignment, utilizing new metrics and validation methods to assess how well translation equivalents share meaning across diverse languages.

Contribution

It presents a novel methodology and metrics for evaluating lexical alignment at the word and domain level, incorporating naturalistic validation and analysis across 16 languages.

Findings

Significant room for improvement with newer language models.

New metrics based on contextualized embeddings show promise.

Analysis highlights the importance of local and domain-specific alignment evaluation.

Abstract

NLP research on aligning lexical representation spaces to one another has so far focused on aligning language spaces in their entirety. However, cognitive science has long focused on a local perspective, investigating whether translation equivalents truly share the same meaning or the extent that cultural and regional influences result in meaning variations. With recent technological advances and the increasing amounts of available data, the longstanding question of cross-lingual lexical alignment can now be approached in a more data-driven manner. However, developing metrics for the task requires some methodology for comparing metric efficacy. We address this gap and present a methodology for analyzing both synthetic validations and a novel naturalistic validation using lexical gaps in the kinship domain. We further propose new metrics, hitherto unexplored on this task, based on contextualized embeddings. Our analysis spans 16 diverse languages, demonstrating that there is substantial room for improvement with the use of newer language models. Our research paves the way for more accurate and nuanced cross-lingual lexical alignment methodologies and evaluation.