Context Vectors are Reflections of Word Vectors in Half the Dimensions

TL;DR

This paper provides a theoretical analysis showing that context vectors in models like word2vec are essentially reflections of word vectors in half the dimensions, based on probabilistic assumptions and matrix analysis.

Contribution

It introduces a theoretical framework linking word and context vectors, demonstrating their relationship through properties of the PMI matrix and proposing a method to tie weights in SGNS.

Findings

Word-word PMI matrix approximates a Gaussian ensemble

Context vectors are reflections of word vectors in half the dimensions

Provides a method for theoretically grounded weight tying in SGNS

Abstract

This paper takes a step towards theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilistic assumptions on the nature of word vectors, context vectors, and text generation. These assumptions are well supported either empirically or theoretically by the existing literature. Next, we show that under these assumptions the widely-used word-word PMI matrix is approximately a random symmetric Gaussian ensemble. This, in turn, implies that context vectors are reflections of word vectors in approximately half the dimensions. As a direct application of our result, we suggest a theoretically grounded way of tying weights in the SGNS model.

Tables

Table 1. Table 1: Corpus statistics. T = 𝑇 absent T= total length in tokens; | 𝒲 | = 𝒲 absent |\mathcal{W}|= number of unique words.

Data set	Size	$T$	$\left|𝒲\right|$
text8	100 MB	17M	254K
enwik9	715 MB	124M	833K

Table 2. Table 2: Evaluation of word embeddings in the analogy tasks (Google and MSR) and in the similarity tasks (the rest). For word similarities evaluation metric is the Spearman’s correlation with the human ratings, while for word analogies it is the percentage of correct answers. Model sizes are in number of trainable parameters.

Data	Model	Size	?	?	?	?	Google	MSR
			WordSim	MEN	M. Turk	Rare Words
text8	SGNS	28M	.681	.241	.631	.072	.307	.286
	SGNS+WT	14M	.637	.215	.624	.057	.314	.319
enwik9	SGNS	87M	.671	.268	.662	.213	.558	.410
	SGNS+WT	44M	.633	.236	.639	.175	.516	.429

Equations

$${\mathbf{w}}_i\stackrel{\text{iid}}{\sim }\mathcal{N}\left(0,\frac{1}{d}\mathbf{I}\right),$$

$${\mathbf{c}}_i={\mathbf{Qw}}_i,i=1,\dots ,n,$$

$$p(i\mid j)\propto p_i\cdot e^{{\mathbf{w}}_j^{\top }{\mathbf{c}}_i}$$

$$p_i\propto \frac{1}{r_i^{1-\alpha }},\alpha \in (0,1].$$

$$\mathrm{Pr}\left(\left|{\mathbf{w}}^{\top }\mathbf{Qw}-\mathrm{Tr}(\mathbf{Q}){\sigma }^2\right|>t\sqrt{2d}{\sigma }^2\right)\le \frac{1}{t^2}.$$

$$E[X]=E\left[\sum _{i,j}{q}_{ij}{w}_i{w}_j\right]=\sum _i{q}_{ii}{\sigma }^2=\mathrm{Tr}(\mathbf{Q}){\sigma }^2$$

$$(E[X]{)}^2=\sum _{i,j}{q}_{ii}{q}_{jj}{\sigma }^4$$

$$E\left[X^2\right]=E\left[{\left(\sum _{i,j}{q}_{ij}{w}_i{w}_j\right)}^2\right]=E\left[\sum _{i,j}{q}_{ii}{q}_{jj}{w}_i^2{w}_j^2+\sum _{i\ne j}(q_{ij}^2+q_{ij}{q}_{ji})w_i^2{w}_j^2\right]$$

$$E\left[X^2\right]$$

$$=\sum _{i\ne j}{q}_{ii}{q}_{jj}{\sigma }^4+3\sum _i{q}_{ii}^2{\sigma }^4+\sum _{i\ne j}(q_{ij}^2+q_{ij}{q}_{ji}){\sigma }^4$$

$$=\sum _{i,j}{q}_{ii}{q}_{jj}{\sigma }^4+2\sum _i{q}_{ii}^2{\sigma }^4+\sum _{i\ne j}(q_{ij}^2+q_{ij}{q}_{ji}){\sigma }^4$$

$$\mathrm{Var}[X]=E\left[X^2\right]-(E[X]{)}^2=2\sum _i{q}_{ii}^2{\sigma }^4+\sum _{i\ne j}(q_{ij}^2+q_{ij}{q}_{ji}){\sigma }^4$$

$$\mathrm{Var}[X]=\left(d+\sum _i{q}_{ii}^2+\sum _{i\ne j}{q}_{ij}{q}_{ji}\right){\sigma }^4=\left(d+\mathrm{Tr}({\mathbf{Q}}^2)\right){\sigma }^4$$

$$\mathrm{Pr}\left(\left|{\mathbf{w}}^{\top }\mathbf{Qw}-\mathrm{Tr}(\mathbf{Q}){\sigma }^2\right|>t\sqrt{2d}{\sigma }^2\right)\le \frac{\left(d+\mathrm{Tr}({\mathbf{Q}}^2)\right){\sigma }^4}{t^2\cdot 2d{\sigma }^4}$$

$$\mathrm{Pr}\left(\left|{\mathbf{w}}^{\top }\mathbf{Qw}-\frac{\mathrm{Tr}(\mathbf{Q})}{d}\right|>t\sqrt{\frac{2}{d}}\right)\le \frac{1}{t^2}.$$

$$\mathrm{Pr}\left(\left|\Vert \mathbf{w}{\Vert }^2-1\right|>t\sqrt{\frac{2}{d}}\right)\le \frac{1}{t^2}.$$

$$Z_j\approx 1+\frac{1}{2d}(\forall j)$$

$$E\left[Z_j\mid {\mathbf{w}}_j\right]$$

$$\displaystyle=p_{j}\operatorname{E}\left[e^{\mathbf{w}_{j}^{\top}\mathbf{Qw}_{j}}\bigm{|}\mathbf{w}_{j}\right]+\sum_{i\neq j}p_{i}M_{\mathbf{c}_{i}}(\mathbf{w}_{j})=p_{j}e^{\mathbf{w}_{j}^{\top}\mathbf{Qw}_{j}}+\sum_{i\neq j}p_{i}e^{\frac{1}{2}\mathbf{w}_{j}^{\top}(\frac{1}{d}\mathbf{I})\mathbf{w}_{j}}$$

$$=p_j{e}^{{\mathbf{w}}_j^{\top }{\mathbf{Qw}}_j}+e^{\frac{1}{2d}\Vert {\mathbf{w}}_j{\Vert }^2}\sum _{i\ne j}{p}_i=p_j\left(e^{{\mathbf{w}}_j^{\top }{\mathbf{Qw}}_j}-e^{\frac{1}{2d}\Vert {\mathbf{w}}_j{\Vert }^2}\right)+e^{\frac{1}{2d}\Vert {\mathbf{w}}_j{\Vert }^2}$$

$${\mathbf{w}}_j^{\top }{\mathbf{Qw}}_j$$

$$\Vert {\mathbf{w}}_j{\Vert }^2$$

$$E\left[Z_j\mid {\mathbf{w}}_j\right]\approx p_j\left(e^{\frac{\mathrm{Tr}(\mathbf{Q})}{d}}-e^{\frac{1}{2d}}\right)+1+\frac{1}{2d}.$$

$$H_{n,\alpha }=\sum _{k=1}^n\frac{1}{k^{1-\alpha }}\sim {\int }_1^n\frac{dx}{x^{1-\alpha }}\sim \frac{n^{\alpha }}{\alpha },$$

$$p_j\sim \frac{\alpha }{r_j^{1-\alpha }{n}^{\alpha }}\to 0\text{ as }n\to \infty .$$

$$E\left[Z_j\mid {\mathbf{w}}_j\right]\approx 1+\frac{1}{2d}.$$

$$\operatorname{Var}\left[Z_{j}\mid\mathbf{w}_{j}\right]=\sum_{i=1}^{n}p_{i}^{2}\operatorname{Var}\left[e^{\mathbf{w}_{j}^{\top}\mathbf{c}_{i}}\bigm{|}\mathbf{w}_{j}\right]+\sum_{i\neq k}p_{i}p_{k}\operatorname{Cov}\left[e^{\mathbf{w}_{j}^{\top}\mathbf{c}_{i}},\,e^{\mathbf{w}_{j}^{\top}\mathbf{c}_{k}}\bigm{|}\mathbf{w}_{j}\right]$$

$$\operatorname{Var}\left[e^{\mathbf{w}_{j}^{\top}\mathbf{c}_{j}}\bigm{|}\mathbf{w}_{j}\right]=\operatorname{Var}\left[e^{\mathbf{w}_{j}^{\top}\mathbf{Qw}_{j}}\bigm{|}\mathbf{w}_{j}\right]=0,$$

$$\displaystyle\operatorname{Var}\left[e^{\mathbf{w}_{j}^{\top}\mathbf{c}_{i}}\bigm{|}\mathbf{w}_{j}\right]$$

$$=M_{{\mathbf{c}}_i}(2{\mathbf{w}}_j)-{\left(M_{{\mathbf{c}}_i}({\mathbf{w}}_j)\right)}^2$$

$$=e^{\frac{2}{d}\Vert {\mathbf{w}}_j{\Vert }^2}-e^{\frac{1}{d}\Vert {\mathbf{w}}_j{\Vert }^2}.$$

$$\operatorname{Cov}\left[e^{\mathbf{w}_{j}^{\top}\mathbf{c}_{i}},\,e^{\mathbf{w}_{j}^{\top}\mathbf{c}_{k}}\bigm{|}\mathbf{w}_{j}\right]=0,\quad i\neq k.$$

$$\mathrm{Var}[Z_j\mid {\mathbf{w}}_j]=\left(e^{\frac{2}{d}\Vert {\mathbf{w}}_j{\Vert }^2}-e^{\frac{1}{d}\Vert {\mathbf{w}}_j{\Vert }^2}\right)\sum _{i\ne j}{p}_i^2.$$

$$\sum _{i\ne j}{p}_i^2\sim \frac{C}{n^{2\alpha }}{\int }_1^n\frac{dx}{x^{2-2\alpha }}\sim \frac{C}{n^{\min \{1,2\alpha \}}}.$$

Full text

1 Introduction and Main Result

Today word embeddings play an important role in many natural language processing tasks, from predictive language models and machine translation to image annotation and question answering, where they are usually ‘plugged in’ to a larger model. An understanding of their properties is of interest as it may allow the development of better performing embeddings and improved interpretability of models using them. This paper takes a step in this direction.

Notation: We let $\mathbb{R}$ denote the real numbers. Bold-faced lowercase letters ($\mathbf{x}$) denote vectors in Euclidean space, bold-faced uppercase letters ($\mathbf{X}$) denote matrices, plain-faced lowercase letters ($x$) denote scalars, plain-faced uppercase letters ($X$) denote scalar random variables, $\|\cdot\|$ denotes the Euclidean norm: $\|\mathbf{x}\|:=\sqrt{\mathbf{x}^{\top}\mathbf{x}}$, ‘i.i.d.’ stands for ‘independent and identically distributed’. We use the sign $\sim$ to abbreviate the phrase ‘distributed as’, and the sign $\propto$ to abbreviate ‘proportional to’. $\operatorname{Tr}(\mathbf{A})$ is used to denote the trace of a matrix $\mathbf{A}$. $M_{\mathbf{x}}(\mathbf{t})$ is the moment-generating function of a random vector $\mathbf{x}$ at $\mathbf{t}$: $M_{\mathbf{x}}(\mathbf{t})=\operatorname{E}[e^{\mathbf{t}^{\top}\mathbf{x}}]$. $\odot$ is the Hadamard product (element-wise multiplication).

Assuming that words have already been converted into indices, let $\{1,\ldots,n\}$ be a finite vocabulary of words. Following the setup of the widely used word2vec model (?), we will use two vectors per each word $i$:

• •

$\mathbf{w}_{i}$ when $i$ is a center word,

• •

$\mathbf{c}_{i}$ when $i$ is a context word.

We make the following key assumptions in our work.

Assumption 1**.**

A priori word vectors $\mathbf{w}_{1},\ldots,\mathbf{w}_{n}\in\mathbb{R}^{d}$ are i.i.d. draws from isotropic multivariate Gaussian distribution:

[TABLE]

where $\mathbf{I}$ is the $d\times d$ identity matrix.

This is motivated by the work of ? (?), where the ensemble of word vectors consists of i.i.d draws generated by $\mathbf{v}=s\cdot\hat{\mathbf{v}}$, with $\hat{\mathbf{v}}$ being from the spherical Gaussian distribution $\mathcal{N}(\mathbf{0},\mathbf{I})$, and $s$ being a scalar random variable with bounded expectation and range. In their work, the norm $\|\mathbf{v}_{i}\|$ of the word vector for a word $i$ is related to its unigram probability $p(i)$, and to allow a sufficient dynamic range for these probabilities they needed the multiplier $s$. In our work, unigram probabilities are not mapped to vector lengths, and this is why we do not need such multiplier. Direct relationship between word probabilities and word vector norms is also implied by the model of ? (?).

Assumption 2**.**

Context vectors $\mathbf{c}_{1},\ldots,\mathbf{c}_{n}$ are related to word vectors according to

[TABLE]

for some orthogonal matrix $\mathbf{Q}\in\mathbb{R}^{d\times d}$.

This is mainly guided by the work of ? (?), who showed that context vectors in the SGNS model of ? (?) are distributed similarly to word vectors in the sense that pairwise cosine distances between word (input) embeddings strongly correlate with the corresponding pairwise cosine distances between context (output) embeddings (see their Table 4). This is why we choose the transform from word vectors to context vectors to be orthogonal as it preserves inner products and consequently Euclidean norms. Notice, that $\mathbf{c}_{i}\,\,{\stackrel{{\scriptstyle\text{iid}}}{{\sim}}}\,\,\mathcal{N}\left(\mathbf{0},\frac{1}{d}\mathbf{I}\right).$

Assumption 3**.**

Given a word $j$, probability of any word $i$ being in its context111Context is a fixed-size symmetric window around the given word. is given by

[TABLE]

where $p_{i}=p(i)$ is the unigram probability for the word $i$, which is inverse proportional to its smoothed frequency rank $r_{i}$, i.e.

[TABLE]

This is similar to the log-linear model of ? (?), but differs in the following aspects: $\mathbf{c}_{i}$ is not assumed to do a random walk over the unit sphere with bounded displacement; we use the factor $p_{i}$ to directly capture word frequencies and do not model them via vector norms. Equation (3) can be interpreted as follows: probability that the word $i$ occurs in the context of the word $j$ is probability that the word $i$ occurs anywhere in a large corpus, corrected for the relationship between words $i$ and $j$. This approach was already considered by ? (?) but in their work $i$ is the entire left context of the word $j$, and $\mathbf{c}_{i}$ is a vector representation of this entire context. Also, like ? (?) but unlike ? (?), we use the model (3) for a theoretical analysis rather than for fitting to data. Smoothing of the unigram probabilities (i.e. raising them to power $1-\alpha$) is motivated by the works of ? (?, ?, ?), where $\alpha=0.25$ is a typical choice. We notice here that $\alpha=0^{+}$ gives us Zipf’s law (?), whereas $\alpha=1$ gives us uniform distribution of word frequencies which is not valid empirically but on the other hand can be used to explain additivity of word vectors (?). The specific value of $\alpha$ is important for

The relationship between word (input) and context (output) vectors was addressed in several previous works. E.g., in recurrent neural network language modeling (RNNLM), tying input and output embeddings is a useful regularization technique introduced earlier (?) and studied in more details recently (?, ?). This technique improves language modeling quality (measured as perplexity of a held-out text) while decreasing the total number of trainable parameters almost two-fold since most of the parameters in RNNLM are due to embedding matrices. The direct application of this regularization technique to SGNS worsens the quality of word vectors as was shown empirically by ? (?) and by ? (?). This worsening was predicted earlier by ? (?) using a simple linguistic observation that words usually do not appear in the contexts of themselves. This basically means that $\mathbf{Q}\neq\mathbf{I}$ in (2). At the same time, there is empirical evidence that the relationship between input and output embeddings is linear (?, ?). In this paper, we provide a theoretical justification for this and reveal the exact form of the transform $\mathbf{Q}$. Our main contribution is the following

Theorem 1**.**

Under Assumptions 1, 2, and 3 above, context vector $\mathbf{c}_{i}$ for a word $i$ is the reflection of the word vector $\mathbf{w}_{i}$ in approximately half of the dimensions.

Figure 1 illustrates this idea for the case $d=2$. In general, our word and context vectors live in a $d$-dimensional vector space over real numbers ($\mathbb{R}^{d}$). By Theorem 1 we can settle them in a $d/2$-dimensional vector space over complex numbers ($\mathbb{C}^{d/2}$) in such way that the context vector $\widetilde{\mathbf{c}}_{i}\in\mathbb{C}^{d/2}$ for a word $i$ is the complex conjugate of the word vector $\widetilde{\mathbf{w}}_{i}\in\mathbb{C}^{d/2}$. This is in line with the Theorem 2 of ? (?), however they use completely different set of basic assumptions and their primary goal is to encode statistical properties of words directly into word vectors.

2 Proof of Theorem 1

The proof is divided into three steps: first we show that the partition function in (3) concentrates around $1$, and thus $\propto$ can be replaced by $\approx$; using this fact we show that $\mathbf{Q}$ is (approximately) an involutary matrix, i.e. similar to $\operatorname{diag}(\{+1,-1\})$; and finally we show that the word-word pointwise-mutual information matrix is approximately symmetric Gaussian random matrix with weakly dependent entries. The latter fact immediately implies the statement of the Theorem 1.

2.1 Concentration of the partition function

We first need the following auxiliary result.

Lemma 1**.**

Let $\mathbf{w}\sim\mathcal{N}(0,\sigma^{2}\mathbf{I})$, then $\forall t>0$ and any orthogonal $\mathbf{Q}$

[TABLE]

Proof.

Consider the random variable $X={\mathbf{w}}^{\top}\mathbf{Q}{\mathbf{w}}$, and let $\mathbf{Q}=[q_{ij}]$. We have

[TABLE]

Therefore,

[TABLE]

Further,

[TABLE]

where we dropped the terms containing odd powers of $w_{i}$, as their expectations are zeros. Hence,

[TABLE]

From (7) and (8) we have

[TABLE]

It is easy to see that $\sum_{i}q_{ii}^{2}+\sum_{i\neq j}q^{2}_{ij}=d$ (the sum of squared elements of an orthogonal matrix). In this way,

[TABLE]

Applying Chebyshev inequality to $X$, and taking into account (6) and (8), we have $\forall\epsilon>0$

[TABLE]

Since $\mathbf{Q}^{2}$ is orthogonal, its trace does not exceed $d$, and we obtain (5). ∎

Remark 1.1*.*

When $\sigma^{2}=\frac{1}{d}$, the inequality (5) becomes:

[TABLE]

Corollary 1.1**.**

Let $\mathbf{w}\sim\mathcal{N}(0,\frac{1}{d}\mathbf{I})$, then $\forall t>0$

[TABLE]

Proof.

The statement follows from Lemma 1 and its Remark 1.1, when $\mathbf{Q}=\mathbf{I}$. ∎

Now we are ready to show that the partition function in (3) concentrates around $1+\frac{1}{2d}$.

Lemma 2**.**

Let $Z_{j}$ be a partition function in (3), i.e. $Z_{j}=\sum_{i=1}^{n}p_{i}e^{\mathbf{w}_{j}^{\top}\mathbf{c}_{i}}$. Then

[TABLE]

Proof.

We will first show that the conditional expectation $\operatorname{E}[Z_{j}\mid\mathbf{w}_{j}]$ depends on $\mathbf{w}_{j}$ mainly through its norm $\|\mathbf{w}_{j}\|$:

[TABLE]

where $M_{\mathbf{c}_{i}}(\mathbf{w}_{j})$ is the moment-generating function of $\mathbf{c}_{i}$ at $\mathbf{w}_{j}$. In Lemma 1 and Corollary 1.1 it is shown that $\mathbf{w}_{j}^{\top}\mathbf{Qw}_{j}$ and $\|\mathbf{w}_{j}\|^{2}$ concentrate well around their means $\operatorname{Tr}(\mathbf{Q})\frac{1}{d}$ and $1$ respectively and thus we can approximate

[TABLE]

The quantity $\frac{1}{2d}$ is small for $d\geq 50$ which is typical for dimensionality of word vectors (?). Thus, using (14), (15), and Maclaurin expansion for $x\mapsto e^{x}$ in the last term of (13), we obtain

[TABLE]

This approximation is very helpful as the right-hand side does not contain $\mathbf{w}_{j}$ and thus it is an approximation for the $\operatorname{E}[Z_{j}]$ as well. Let $H_{n,\alpha}$ be the normalizer in (4), then

[TABLE]

and thus we have222We abuse the notation here and use ‘$\sim$’ to denote asymptotic equivalence, i.e. $f(n)\sim g(n)$ iff $\lim_{n\to\infty}\frac{f(n)}{g(n)}=1$.

[TABLE]

Now, combining (16), (17), and (15), we get

[TABLE]

Now let us show that $\operatorname{Var}[Z_{j}]$ is small relative to the mean $\operatorname{E}[Z_{j}]$. First, we have

[TABLE]

For the variance terms we have

[TABLE]

and

[TABLE]

Conditioned on $\mathbf{w}_{j}$, the random variables $\{e^{\mathbf{w}_{j}^{\top}\mathbf{c}_{i}}\}_{i\neq j}$ are independent, while $e^{\mathbf{w}_{j}^{\top}\mathbf{c}_{j}}$ is constant, and thus

[TABLE]

From (19), (20), (21), and (22), we have

[TABLE]

Notice that for some constant $C>0$

[TABLE]

Combining (15), (23), (24), and using Maclaurin expansion for $x\mapsto e^{x}$, we have

[TABLE]

Now, the statement of the lemma follows from (18) and (25). ∎

Remark 2.1*.*

Lemma 2 basically says that under the Assumptions 1 and 2, the model (3) self-normalizes, i.e. the normalization term is almost constant and moreover it is almost $1$. This result is similar to the result of ? (?), but differs in that our model (3) is not log-linear as its condition ($j$) and prediction ($i$) are both parameterized. The result of ? (?) on self-normalization of the NCE language models is closer to ours but the setup differs in that $p_{i}$ does not appear as a factor in their model. We finally notice that Lemma 2 is an analogue of Lemma 2.1 from ? (?) but adapted to our settings.

2.2 $\mathbf{Q}$ is an involutary matrix

Lemma 3**.**

Let $\mathbf{Q}$ be the matrix mapping word vectors to context vectors as in (2). Then, under Assumptions 1, 2, and 3, $\mathbf{Q}$ is approximately an involutary matrix.

Proof.

The dimensionality $d$ of word vectors is usually in the range $[50,1000]$ (?), and thus we can neglect the term $\frac{1}{2d}$ in (12) and approximate $Z_{j}\approx 1$. This means that the model (3) simplifies to

[TABLE]

or, equivalently

[TABLE]

where $p(i,\,j)$ is the probability that the words $i$ and $j$ co-occur in the same context window. Notice that the left-hand side in (26) is the pointwise mutual information (PMI) between words $i$ and $j$. From (2) and (26) we have

[TABLE]

where $\operatorname{PMI}$ stands for the PMI-matrix, and $\mathbf{W}$ is a $n\times d$ matrix in which $i$-th row is $\mathbf{w}_{i}^{\top}$. Since $p(i,j)=p(j,i)$, we should have $\operatorname{PMI}=\operatorname{PMI}^{\top}$, which implies

[TABLE]

where we used the fact that $\mathbf{W}^{\top}\mathbf{W}\approx\frac{1}{d}\mathbf{I}$. Since $\mathbf{Q}$ is assumed to be orthogonal, from (27) we get

[TABLE]

Thus, $\mathbf{Q}$ is approximately an involutary matrix, and we can choose it to be a signature matrix, i.e. a diagonal matrix with $\pm 1$ on the diagonal333One can show that any involutary matrix can be represented as $\mathbf{P}^{\top}\operatorname{diag}(\pm 1,\ldots,\pm 1)\mathbf{P}$, where $\mathbf{P}$ is orthogonal, and thus by reparametrization $\widetilde{\mathbf{w}}_{i}=\mathbf{P}\mathbf{w}_{i}\,\,{\stackrel{{\scriptstyle\text{iid}}}{{\sim}}}\,\,\mathcal{N}(\mathbf{0},\frac{1}{d}\mathbf{I})$, we can still have (28).:

[TABLE]

∎

In this way, context vectors are word vectors with some of the coordinates being multiplied by $-1$. The natural question is: how many of the coordinates should be “flipped”?

2.3 PMI as a random matrix

Let $\mathbf{x}_{i}\in\mathbb{R}^{l}$ be the vector consisting of the first $l$ coordinates of $\mathbf{w}_{i}$, i.e.

[TABLE]

and let $\mathbf{y}_{i}\in\mathbb{R}^{d-l}$ be the vector consisting of the last $d-l$ coordinates of $\mathbf{w}_{i}$, i.e.

[TABLE]

Due to Assumption (1), $\mathbf{x}_{i}$’s are i.i.d. draws from $\mathcal{N}\left(0,\frac{1}{d}\mathbf{I}_{l\times l}\right)$, $\mathbf{y}_{i}$’s are i.i.d. draws from $\mathcal{N}\left(0,\frac{1}{d}\mathbf{I}_{(d-l)\times(d-l)}\right)$, and $\{\mathbf{x}_{i}\}$, $\{\mathbf{y}_{i}\}$ are jointly independent. Without restricting the generality, assume that the first $l$ diagonal elements in (28) are equal to $+1$, and the rest $d-l$ elements are equal to $-1$. Thus

[TABLE]

For $i\neq j$, $\mathbf{x}_{i}^{\top}\mathbf{x}_{j}$ is a sum of $l$ i.i.d. random variables with mean 0 and variance $\frac{1}{d^{2}}$, and by Central Limit Theorem, $\mathbf{x}_{i}^{\top}\mathbf{x}_{j}\approx\mathcal{N}\left(0,\frac{l}{d^{2}}\right)$. Similarly, $\mathbf{y}_{i}^{\top}\mathbf{y}_{j}\approx\mathcal{N}\left(0,\frac{d-l}{d^{2}}\right)$, and thus

[TABLE]

For $i=j$, we have

[TABLE]

where $\chi^{2}_{l}$ is a chi-square random variable with $l$ degrees of freedom. By combinatorial argument (similar to that of Lemma 1) one can show that covariance between any two distinct and non-symmetric entries of $\mathbf{WQ}^{\top}\mathbf{W}^{\top}$ is zero, and thus

[TABLE]

Moreover, we can show that $\operatorname{PMI}_{ij}$ and $\operatorname{PMI}_{pq}$ tend to be independent when $d$ is large enough. For the case $i\neq p$, $j\neq q$ this follows directly from the Assumption (1). Now consider the case $i=p$, $j\neq q$ (two distinct elements from the same row). The case $i\neq p$, $j=q$ (two distinct elements from the same column) can be analyzed similarly. Let $\mathbf{t}=\begin{bmatrix}t_{1}&t_{2}\end{bmatrix}^{\top}$. Then the moment-generating function (m.g.f.) of $\begin{bmatrix}\operatorname{PMI}_{ij}&\operatorname{PMI}_{iq}\end{bmatrix}^{\top}$ at $\mathbf{t}$ is

[TABLE]

and the last expression for large $d$ is approximately $e^{\frac{\|\mathbf{t}\|^{2}}{2d}}$ which is the m.g.f. of a two-dimensional Gaussian vector with distribution $\mathcal{N}(\mathbf{0},\frac{1}{d}\mathbf{I}_{2\times 2})$. Hence

[TABLE]

which implies approximate independence between $\operatorname{PMI}_{ij}$ and $\operatorname{PMI}_{iq}$. Hence, from (31) and (32) we conclude that for the PMI matrix

• •

the above-diagonal entries have (approximately) distribution $\mathcal{N}\left(0,\frac{1}{d}\right)$,

• •

the diagonal entries have (approximately) distribution $\mathcal{N}\left(\frac{2l-d}{d},\frac{2}{d}\right)$,

• •

all entries on and above its diagonal tend to pairwise independence.

This means that the PMI matrix is an approximately symmetric Gaussian random matrix with weakly dependent entries and it is known that the empirical distribution of eigenvalues of such matrix approaches symmetric around [math] distribution as its size $n$ increases (?).444Usually papers on random matrix theory do not state this explicitly. Rather they show that the limiting distribution has odd moments equal to 0. This is the case for the referenced paper as well, see the proof of their Theorem 2.2. Thus, we should have

[TABLE]

Recall, that $d\|\mathbf{x}_{i}\|^{2}\sim\chi^{2}_{l}$ and $d\|\mathbf{y}_{i}\|^{2}\sim\chi^{2}_{d-l}$. Hence, taking expectation on both sides of (36) we have

[TABLE]

which concludes the proof of Theorem 1.

Remark 3.1*.*

In terms of the introduced notation (29) and (30), each word’s vector $\mathbf{w}_{i}$ splits into two subvectors $\mathbf{x}_{i}$ and $\mathbf{y}_{i}$, and due to Theorem 1, our model (3) for generating a word $i$ in the context of a word $j$ can be rewritten as

[TABLE]

Interestingly, embeddings of the first type ($\mathbf{x}_{i}$ and $\mathbf{x}_{j}$) are responsible for pulling the word $i$ into the context of the word $j$, while embeddings of the second type ($\mathbf{y}_{i}$ and $\mathbf{y}_{j}$) are responsible for pushing the word $i$ away from the context of the word $j$. We hypothesize that the $\mathbf{x}$-embeddings are more related to semantics, whereas the $\mathbf{y}$-embeddings are more related to syntax. We defer testing of this hypothesis to our future work.

3 Empirical verification

To verify that the real-world PMI matrices have indeed symmetric (around 0) distribution of their eigenvalues, we consider two widely-used datasets, text8 and enwik9,555http://mattmahoney.net/dc/textdata.html. The enwik9 data was processed with the Perl-script wikifil.pl provided on the same webpage. It filters Wikipedia XML dumps to “clean” text consisting only of lowercase letters and spaces (never consecutive). from which we extract PMI matrices using the hyperwords tool of ? (?). We use the default settings for all hyperparameters, except word frequency threshold and context window size. We were ignoring words that appeared less than 100 times and 150 times in text8 and enwik9 correspondingly, resulting in vocabularies of 11,815 and 29,145 correspondingly. We additionally experiment with context window 5, which by default is set to 2, and which we believe could affect the results. The eigenvalues of the PMI matrices are then calculated using the TensorFlow library (?), and the above-mentioned threshold of 150 for enwik9 was chosen to fit the resulting PMI matrix into the GPU memory (12GB, NVIDIA Titan X Maxwell). The histograms of eigenvalues are provided in Figure 2.

As we can see, the distributions are not perfectly symmetric with a little right skewness, but in general they seem to be symmetric. Notice, that this is in stark contrast with the equation (2.5) from ? (?), which claims that the PMI matrix should be approximately positive semi-definite, i.e. that it should have mostly positive eigenvalues. Also, notice that the shapes of distributions are far from resembling the Wigner semicircle law $x\mapsto\frac{1}{2\pi}\sqrt{4-x^{2}}$, which is the limiting distribution for the eigenvalues of many random symmetric matrices with i.i.d. entries (?, ?). This means that the entries of a typical PMI matrix are dependent, otherwise we would observe approximately semicircle distributions for its eigenvalues. Interestingly, there is a striking similarity between the shapes of distributions in Figure 2 and of spectral densities of the scale-free random graphs (?) and random graphs with expected degrees (?) which arise in physics and network science. Notice that the connection between human language structure and scale-free random graphs was observed previously by ? (?), and it would be interesting to dive deeper in this direction.

4 Weight tying in skip-gram model

We would like to apply our results to tie embeddings in the skip-gram model of ? (?) in a theoretically grounded way. One may argue that our key Assumption 3 differs from the softmax-prediction of the skip-gram model. Although this is true, in fact the softmax normalization is never used in practice when training skip-gram. Instead it is common to replace the softmax cross-entropy by the negative sampling objective (equation (4) in ? (?)), and its optimization is almost equivalent to finding a low-rank approximation of the shifted word-word PMI matrix in the form $\mathbf{w}_{i}^{\top}\mathbf{c}_{j}\approx\operatorname{PMI}_{ij}-\log k$ (?). Since our Assumptions lead to the same conclusion up to a constant shift (26), we believe that Theorem 1 can be directly applied to tie word ($\mathbf{w}_{i}$) and context ($\mathbf{c}_{i}$) embeddings in the SGNS model. For this purpose we form a vector $\mathbf{q}\in\mathbb{R}^{d}$ of $d$ i.i.d. draws from the Rademacher distribution666Rademacher distribution is a discrete probability distribution where a random variate $X$ has a 50% chance of being $+1$ and a 50% chance of being $-1$. and then put

[TABLE]

for all words $i$ in the vocabulary. This is equivalent to (2) when the matrix $\mathbf{Q}$ has the special diagonal form (28). Such modification of the SGNS is refered to as ‘SGNS + WT’. The word embeddings $\mathbf{w}_{i}$ are initialized randomly, and then trained on text8 and enwik9 using the reference word2vec implementation from the TensorFlow codebase777https://github.com/tensorflow/models/blob/master/tutorials/embedding/word2vec.py with all hyperparameters set to their default values except that we choose the learning rate to decay 20% faster in the weight-tied model. This additional tuning of the learning rate decay is not surprising: the model with tied embeddings has two times less parameters compared to the model with untied weights, and this leads to a significant change of the optimization landscape, which in turn results in the need to tune the most sensitive hyperparameter — the learning rate (or its decay schedule). As is standard nowadays the trained embeddings are evaluated on several word similarity and word analogy tasks. We used the hyperwords tool of ? (?) and we refer the reader to their paper for the methodology of evaluation. We only mention here few key points:

• •

Our goal is not to beat state of the art, but to empirically validate the statement of Theorem 1. This is why we were evaluating only word (input) embeddings for both SGNS and SGNS+WT. I.e., we were not adding context vectors to word vectors in the similarity tasks, as it is usually done nowadays.

• •

Word similarity datasets contain word pairs together with human-assigned similarity scores. The word vectors are evaluated by ranking the pairs according to their cosine similarities and measuring the correlation (Spearman’s $\rho$) with the human ratings.

• •

For answering analogy questions ($a$ is to $b$ as $c$ is to $?$) we use the 3CosMul of ? (?) and the evaluation metric for the analogy questions is the percentage of correct answers.

The results of evaluation are provided in Table 2. As we can see, SGNS + WT produces embeddings comparable in quality with those produced by the baseline SGNS model despite having 50% fewer parameters. This also empirically validates the statement of our Theorem 1. We notice that similar results can be obtained by letting the linear transform $\mathbf{Q}$ be a trainable matrix as shown by ? (?). The main difference of our approach is that we know exactly the form of $\mathbf{Q}$, and thus we do not need to learn it.

5 Conclusion

There is a remarkable relationship between human language and other branches of science, and we can get interesting and practical results by studying deeper such relationships. For example, the modern theory of random matrices is replete with theoretical results that can be immediately applied to models of natural language once such models are cast into the appropriate probabilistic setting, as is done in this paper.