Distributional Semantics, Holism, and the Instability of Meaning

TL;DR

This paper investigates the stability of distributional semantic models, distinguishing between types of instability, and argues that their inherent differential instability supports meaningful language change without undermining communication.

Contribution

The paper clarifies the concept of instability in distributional models, introduces the idea of differential instability, and shows how it allows for stable yet adaptable language meaning.

Findings

Differential instability involves relative, not absolute, changes in word relationships.

Models exhibit instability as corpus size increases, influenced by word frequency and consistency.

Differential instability enables meaningful language change without systemic breakdown.

Abstract

Large Language Models are built on the so-called distributional semantic approach to linguistic meaning that has the distributional hypothesis at its core. The distributional hypothesis involves a holistic conception of word meaning: the meaning of a word depends upon its relations to other words in the model. A standard objection to holism is the charge of instability: any change in the meaning properties of a linguistic system (a human speaker, for example) would lead to many changes or a complete change in the entire system. We examine whether the instability objection poses a problem for distributional models of meaning. First, we distinguish between distinct forms of instability that these models could exhibit, and argue that only one such form is relevant for understanding the relation between instability and communication: what we call differential instability. Differential instability is variation in the relative distances between points in a space, rather than variation in the absolute position of those points. We distinguish differential and absolute instability by constructing two of our own smaller language models. We demonstrate the two forms of instability by showing these models change as the corpora they are constructed from increase in size. We argue that the instability that these models display is constrained by the structure and scale of relationships between words, such that the resistance to change for a word is roughly proportional to its frequent and consistent use within the language system. The differential instability that language models exhibit allows for productive forms of meaning change while not leading to the problems raised by the instability objection.