# Context Vectors are Reflections of Word Vectors in Half the Dimensions

## TL;DR

This paper provides a theoretical analysis showing that context vectors in models like word2vec are essentially reflections of word vectors in half the dimensions, based on probabilistic assumptions and matrix analysis.

## Contribution

It introduces a theoretical framework linking word and context vectors, demonstrating their relationship through properties of the PMI matrix and proposing a method to tie weights in SGNS.

## Key findings

- Word-word PMI matrix approximates a Gaussian ensemble
- Context vectors are reflections of word vectors in half the dimensions
- Provides a method for theoretically grounded weight tying in SGNS

## Abstract

This paper takes a step towards theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilistic assumptions on the nature of word vectors, context vectors, and text generation. These assumptions are well supported either empirically or theoretically by the existing literature. Next, we show that under these assumptions the widely-used word-word PMI matrix is approximately a random symmetric Gaussian ensemble. This, in turn, implies that context vectors are reflections of word vectors in approximately half the dimensions. As a direct application of our result, we suggest a theoretically grounded way of tying weights in the SGNS model.