AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples

TL;DR

AM2iCo is a comprehensive multilingual evaluation dataset designed to assess how well state-of-the-art models understand word meaning in context across diverse languages, highlighting significant gaps especially in low-resource and dissimilar languages.

Contribution

The paper introduces AM2iCo, a new evaluation set that overcomes limitations of existing datasets by covering more languages and providing adversarial examples for better model diagnostics.

Findings

Current models underperform humans in understanding word meaning in context.

Performance gaps are largest for low-resource and dissimilar languages.

AM2iCo reveals the need for improved multilingual and cross-lingual models.

Abstract

Capturing word meaning in context and distinguishing between correspondences and variations across languages is key to building successful multilingual and cross-lingual text representation models. However, existing multilingual evaluation datasets that evaluate lexical semantics "in-context" have various limitations. In particular, 1) their language coverage is restricted to high-resource languages and skewed in favor of only a few language families and areas, 2) a design that makes the task solvable via superficial cues, which results in artificially inflated (and sometimes super-human) performances of pretrained encoders, on many target languages, which limits their usefulness for model probing and diagnostics, and 3) little support for cross-lingual evaluation. In order to address these gaps, we present AM2iCo (Adversarial and Multilingual Meaning in Context), a wide-coverage cross-lingual and multilingual evaluation set; it aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts for 14 language pairs. We conduct a series of experiments in a wide range of setups and demonstrate the challenging nature of AM2iCo. The results reveal that current SotA pretrained encoders substantially lag behind human performance, and the largest gaps are observed for low-resource languages and languages dissimilar to English.