Discovering linguistic (ir)regularities in word embeddings through max-margin separating hyperplanes

TL;DR

This paper introduces a novel max-margin hyperplane method to identify linguistic regularities in word embeddings, revealing that related words are orthogonal rather than parallel in embedding space, and demonstrates robustness across various models.

Contribution

The paper proposes a new approach using max-margin hyperplanes to better capture linguistic relationships in word embeddings, improving over previous offset-based methods.

Findings

Related words are closer to orthogonal than parallel in embedding space.

Max-margin hyperplanes effectively represent relationships between words.

The method is robust across different embedding algorithms and context choices.

Abstract

We experiment with new methods for learning how related words are positioned relative to each other in word embedding spaces. Previous approaches learned constant vector offsets: vectors that point from source tokens to target tokens with an assumption that these offsets were parallel to each other. We show that the offsets between related tokens are closer to orthogonal than parallel, and that they have low cosine similarities. We proceed by making a different assumption; target tokens are linearly separable from source and un-labeled tokens. We show that a max-margin hyperplane can separate target tokens and that vectors orthogonal to this hyperplane represent the relationship between source and targets. We find that this representation of the relationship obtains the best results in dis-covering linguistic regularities. We experiment with vector space models trained by a variety of algorithms (Word2vec: CBOW/skip-gram, fastText, or GloVe), and various word context choices such as linear word-order, syntax dependency grammars, and with and without knowledge of word position. These experiments show that our model, SVMCos, is robust to a range of experimental choices when training word embeddings.