Lacking the embedding of a word? Look it up into a traditional dictionary

TL;DR

This paper introduces two methods, DefiNNet and DefBERT, that leverage traditional dictionary definitions to generate embeddings for rare or out-of-vocabulary words, outperforming existing approaches.

Contribution

The paper proposes novel methods using dictionary definitions to produce embeddings for rare words, significantly improving over state-of-the-art techniques.

Findings

DefiNNet outperforms FastText for rare words.

DefBERT outperforms BERT for out-of-vocabulary words.

Dictionary definitions are effective for embedding rare words.

Abstract

Word embeddings are powerful dictionaries, which may easily capture language variations. However, these dictionaries fail to give sense to rare words, which are surprisingly often covered by traditional dictionaries. In this paper, we propose to use definitions retrieved in traditional dictionaries to produce word embeddings for rare words. For this purpose, we introduce two methods: Definition Neural Network (DefiNNet) and Define BERT (DefBERT). In our experiments, DefiNNet and DefBERT significantly outperform state-of-the-art as well as baseline methods devised for producing embeddings of unknown words. In fact, DefiNNet significantly outperforms FastText, which implements a method for the same task-based on n-grams, and DefBERT significantly outperforms the BERT method for OOV words. Then, definitions in traditional dictionaries are useful to build word embeddings for rare words.