Probabilistic Embeddings with Laplacian Graph Priors

TL;DR

This paper presents Probabilistic Embeddings with Laplacian Priors (PELP), a unified framework that incorporates graph information into static word embeddings, generalizing and combining various existing methods while demonstrating flexibility and empirical effectiveness.

Contribution

The paper introduces PELP, a novel probabilistic embedding model that unifies and extends multiple existing embedding techniques using Laplacian graph priors.

Findings

PELP matches the performance of existing models in standard tasks.

PELP can be applied to analyze political sociolects over time.

The authors provide a TensorFlow implementation for flexible use.

Abstract

We introduce probabilistic embeddings using Laplacian priors (PELP). The proposed model enables incorporating graph side-information into static word embeddings. We theoretically show that the model unifies several previously proposed embedding methods under one umbrella. PELP generalises graph-enhanced, group, dynamic, and cross-lingual static word embeddings. PELP also enables any combination of these previous models in a straightforward fashion. Furthermore, we empirically show that our model matches the performance of previous models as special cases. In addition, we demonstrate its flexibility by applying it to the comparison of political sociolects over time. Finally, we provide code as a TensorFlow implementation enabling flexible estimation in different settings.