Word Representations via Gaussian Embedding

TL;DR

This paper proposes a novel approach to word embeddings using Gaussian distributions instead of point vectors, offering advantages in modeling uncertainty, asymmetry, and complex relationships.

Contribution

It introduces a method for learning Gaussian-based word embeddings and demonstrates their effectiveness on benchmarks and in modeling asymmetric semantic relationships.

Findings

Gaussian embeddings outperform point vectors on standard benchmarks

They better capture asymmetric relationships like entailment

The approach provides richer, more expressive representations

Abstract

Current work in lexical distributed representations maps each word to a point vector in low-dimensional space. Mapping instead to a density provides many interesting advantages, including better capturing uncertainty about a representation and its relationships, expressing asymmetries more naturally than dot product or cosine similarity, and enabling more expressive parameterization of decision boundaries. This paper advocates for density-based distributed embeddings and presents a method for learning representations in the space of Gaussian distributions. We compare performance on various word embedding benchmarks, investigate the ability of these embeddings to model entailment and other asymmetric relationships, and explore novel properties of the representation.