Morphological Priors for Probabilistic Neural Word Embeddings

TL;DR

This paper introduces a probabilistic framework that integrates morphological priors with distributional data to enhance word embeddings, especially for rare or unseen words, improving both similarity measures and POS tagging.

Contribution

It presents a novel unified probabilistic model combining morphological and distributional information for word embeddings, unlike prior methods that use only morphemes.

Findings

Improved intrinsic word similarity scores.

Enhanced performance in part-of-speech tagging.

Effective handling of rare and unseen words.

Abstract

Word embeddings allow natural language processing systems to share statistical information across related words. These embeddings are typically based on distributional statistics, making it difficult for them to generalize to rare or unseen words. We propose to improve word embeddings by incorporating morphological information, capturing shared sub-word features. Unlike previous work that constructs word embeddings directly from morphemes, we combine morphological and distributional information in a unified probabilistic framework, in which the word embedding is a latent variable. The morphological information provides a prior distribution on the latent word embeddings, which in turn condition a likelihood function over an observed corpus. This approach yields improvements on intrinsic word similarity evaluations, and also in the downstream task of part-of-speech tagging.