Mimicking Word Embeddings using Subword RNNs

TL;DR

This paper introduces MIMICK, a novel method that generates embeddings for out-of-vocabulary words using subword RNNs, improving NLP task performance without retraining on original corpora.

Contribution

MIMICK is a new approach that composes OOV word embeddings from spellings, avoiding the need for re-training on the original embedding corpus.

Findings

Improves POS and morphosyntactic tagging across 23 languages.

Outperforms baseline word-based methods in OOV scenarios.

Competitive with supervised character-based models in low-resource settings.

Abstract

Word embeddings improve generalization over lexical features by placing each word in a lower-dimensional space, using distributional information obtained from unlabeled data. However, the effectiveness of word embeddings for downstream NLP tasks is limited by out-of-vocabulary (OOV) words, for which embeddings do not exist. In this paper, we present MIMICK, an approach to generating OOV word embeddings compositionally, by learning a function from spellings to distributional embeddings. Unlike prior work, MIMICK does not require re-training on the original word embedding corpus; instead, learning is performed at the type level. Intrinsic and extrinsic evaluations demonstrate the power of this simple approach. On 23 languages, MIMICK improves performance over a word-based baseline for tagging part-of-speech and morphosyntactic attributes. It is competitive with (and complementary to) a supervised character-based model in low-resource settings.