Theoretical foundations and limits of word embeddings: what types of meaning can they capture?

TL;DR

This paper explores the theoretical foundations and limitations of word embeddings in modeling meaning, analyzing their assumptions and potential to clarify core concepts in cultural sociology.

Contribution

It formalizes the relationship between word embeddings and linguistic theories of meaning, highlighting their strengths and vulnerabilities.

Findings

Word embeddings model relational and coherent aspects of meaning.

They are vulnerable to critiques of static and relational assumptions.

Formalization offers new avenues for conceptual clarity in cultural sociology.

Abstract

Measuring meaning is a central problem in cultural sociology and word embeddings may offer powerful new tools to do so. But like any tool, they build on and exert theoretical assumptions. In this paper I theorize the ways in which word embeddings model three core premises of a structural linguistic theory of meaning: that meaning is relational, coherent, and may be analyzed as a static system. In certain ways, word embedding methods are vulnerable to the same, enduring critiques of these premises. In other ways, they offer novel solutions to these critiques. More broadly, formalizing the study of meaning with word embeddings offers theoretical opportunities to clarify core concepts and debates in cultural sociology, such as the coherence of meaning. Just as network analysis specified the once vague notion of social relations (Borgatti et al. 2009), formalizing meaning with embedding methods can push us to specify and reimagine meaning itself.