Massively Multilingual Lexical Specialization of Multilingual Transformers

TL;DR

This paper demonstrates that exposing multilingual transformers to large-scale lexical knowledge from BabelNet significantly improves their performance on cross-lingual lexical tasks, including for unseen languages, through a contrastive lexical specialization process.

Contribution

It introduces a large-scale multilingual lexical specialization method for transformers using BabelNet, improving cross-lingual lexical task performance and enabling generalization to unseen languages.

Findings

Substantial gains in bilingual lexicon induction and cross-lingual word similarity.

Improved cross-lingual sentence retrieval performance.

Generalization to languages without lexical constraints.

Abstract

While pretrained language models (PLMs) primarily serve as general-purpose text encoders that can be fine-tuned for a wide variety of downstream tasks, recent work has shown that they can also be rewired to produce high-quality word representations (i.e., static word embeddings) and yield good performance in type-level lexical tasks. While existing work primarily focused on the lexical specialization of monolingual PLMs with immense quantities of monolingual constraints, in this work we expose massively multilingual transformers (MMTs, e.g., mBERT or XLM-R) to multilingual lexical knowledge at scale, leveraging BabelNet as the readily available rich source of multilingual and cross-lingual type-level lexical knowledge. Concretely, we use BabelNet's multilingual synsets to create synonym pairs (or synonym-gloss pairs) across 50 languages and then subject the MMTs (mBERT and XLM-R) to a lexical specialization procedure guided by a contrastive objective. We show that such massively multilingual lexical specialization brings substantial gains in two standard cross-lingual lexical tasks, bilingual lexicon induction and cross-lingual word similarity, as well as in cross-lingual sentence retrieval. Crucially, we observe gains for languages unseen in specialization, indicating that multilingual lexical specialization enables generalization to languages with no lexical constraints. In a series of subsequent controlled experiments, we show that the number of specialization constraints plays a much greater role than the set of languages from which they originate.