Using $k$-way Co-occurrences for Learning Word Embeddings

TL;DR

This paper extends word co-occurrence analysis to $k$-way interactions, providing a theoretical relationship and a new learning objective, leading to embeddings that perform well even with sparse data.

Contribution

It introduces a theoretical framework and a novel learning method for $k$-way co-occurrences in word embeddings, improving performance on various tasks.

Findings

Theoretical relationship between joint probability and embeddings established.

$k$-way embeddings perform comparably or better than 2-way embeddings.

Empirical validation of the theoretical relationship.

Abstract

Co-occurrences between two words provide useful insights into the semantics of those words. Consequently, numerous prior work on word embedding learning have used co-occurrences between two words as the training signal for learning word embeddings. However, in natural language texts it is common for multiple words to be related and co-occurring in the same context. We extend the notion of co-occurrences to cover $k(\geq\!\!2)$-way co-occurrences among a set of $k$-words. Specifically, we prove a theoretical relationship between the joint probability of $k(\geq\!\!2)$ words, and the sum of $\ell_2$ norms of their embeddings. Next, we propose a learning objective motivated by our theoretical result that utilises $k$-way co-occurrences for learning word embeddings. Our experimental results show that the derived theoretical relationship does indeed hold empirically, and despite data sparsity, for some smaller $k$ values, $k$-way embeddings perform comparably or better than $2$-way embeddings in a range of tasks.