Norm of Word Embedding Encodes Information Gain

TL;DR

This paper reveals that the squared norm of static word embeddings encodes the information gain of words, as measured by KL divergence, and extends this insight to contextualized embeddings, providing a new metric for word informativeness.

Contribution

It establishes a theoretical link between embedding norms and information gain, supported by experiments, and introduces a practical metric for word informativeness in NLP tasks.

Findings

Squared norm encodes information gain via KL divergence.

Theoretical framework explains the embedding norm phenomenon.

Embedding norm is useful for keyword and hypernym discrimination.

Abstract

Distributed representations of words encode lexical semantic information, but what type of information is encoded and how? Focusing on the skip-gram with negative-sampling method, we found that the squared norm of static word embedding encodes the information gain conveyed by the word; the information gain is defined by the Kullback-Leibler divergence of the co-occurrence distribution of the word to the unigram distribution. Our findings are explained by the theoretical framework of the exponential family of probability distributions and confirmed through precise experiments that remove spurious correlations arising from word frequency. This theory also extends to contextualized word embeddings in language models or any neural networks with the softmax output layer. We also demonstrate that both the KL divergence and the squared norm of embedding provide a useful metric of the informativeness of a word in tasks such as keyword extraction, proper-noun discrimination, and hypernym discrimination.