Word Embedding with Neural Probabilistic Prior

TL;DR

This paper introduces a probabilistic prior for word embeddings that regularizes learning, enhances robustness, and can be integrated into existing models, leading to improved performance across multiple tasks.

Contribution

It proposes a novel probabilistic prior as a generative model for word embeddings, improving their quality and stability.

Findings

Enhanced word representations in various tasks

Improved robustness and stability of embeddings

Easy integration with existing models

Abstract

To improve word representation learning, we propose a probabilistic prior which can be seamlessly integrated with word embedding models. Different from previous methods, word embedding is taken as a probabilistic generative model, and it enables us to impose a prior regularizing word representation learning. The proposed prior not only enhances the representation of embedding vectors but also improves the model's robustness and stability. The structure of the proposed prior is simple and effective, and it can be easily implemented and flexibly plugged in most existing word embedding models. Extensive experiments show the proposed method improves word representation on various tasks.