Scale-dependent Relationships in Natural Language

TL;DR

This study investigates how different context scales in Word2Vec embeddings capture various linguistic relationships, highlighting the importance of scale in understanding semantic and syntactic structures in language.

Contribution

It systematically analyzes the impact of sampling scale on Word2Vec embeddings, revealing scale-dependent encoding of linguistic relationships and advocating for scale-free models.

Findings

Different scales encode different linguistic relationships.

Neighborhoods of words vary significantly with scale.

Scale influences the types of semantic and syntactic relations captured.

Abstract

Natural language exhibits statistical dependencies at a wide range of scales. For instance, the mutual information between words in natural language decays like a power law with the temporal lag between them. However, many statistical learning models applied to language impose a sampling scale while extracting statistical structure. For instance, Word2Vec constructs a vector embedding that maximizes the prediction between a target word and the context words that appear nearby in the corpus. The size of the context is chosen by the user and defines a strong scale; relationships over much larger temporal scales would be invisible to the algorithm. This paper examines the family of Word2Vec embeddings generated while systematically manipulating the sampling scale used to define the context around each word. The primary result is that different linguistic relationships are preferentially encoded at different scales. Different scales emphasize different syntactic and semantic relations between words.Moreover, the neighborhoods of a given word in the embeddings change significantly depending on the scale. These results suggest that any individual scale can only identify a subset of the meaningful relationships a word might have, and point toward the importance of developing scale-free models of semantic meaning.