Pragmatic Constraint on Distributional Semantics

TL;DR

This paper explores the inherent limitations of language models in statistical learning, revealing how Zipf's law influences token distribution and impacts semantic understanding.

Contribution

It uncovers the fundamental constraints imposed by Zipf's law on distributional semantics and how token properties affect learning processes.

Findings

Zipf-law token distribution appears regardless of tokenization

Tokens with clear semantic correspondence differ statistically from ambiguous tokens

These properties interfere with distributional semantic learning

Abstract

This paper studies the limits of language models' statistical learning in the context of Zipf's law. First, we demonstrate that Zipf-law token distribution emerges irrespective of the chosen tokenization. Second, we show that Zipf distribution is characterized by two distinct groups of tokens that differ both in terms of their frequency and their semantics. Namely, the tokens that have a one-to-one correspondence with one semantic concept have different statistical properties than those with semantic ambiguity. Finally, we demonstrate how these properties interfere with statistical learning procedures motivated by distributional semantics.