Semantic Shift: the Fundamental Challenge in Text Embedding and Retrieval

TL;DR

This paper investigates how semantic evolution within texts causes embedding collapse and retrieval issues in transformer models, offering a theoretical framework and a measurable indicator called semantic shift.

Contribution

It introduces the concept of semantic shift as a key factor in embedding pathology, providing a formal measure and empirical validation across models and datasets.

Findings

Semantic shift correlates with embedding concentration and retrieval performance.

Text length alone does not predict embedding degradation.

Semantic shift can diagnose when anisotropy harms retrieval.

Abstract

Transformer-based embedding models rely on pooling to map variable-length text into a single vector, enabling efficient similarity search but also inducing well-known geometric pathologies such as anisotropy and length-induced embedding collapse. Existing accounts largely describe \emph{what} these pathologies look like, yet provide limited insight into \emph{when} and \emph{why} they harm downstream retrieval. In this work, we argue that the missing causal factor is \emph{semantic shift}: the intrinsic, structured evolution and dispersion of semantics within a text. We first present a theoretical analysis of \emph{semantic smoothing} in Transformer embeddings: as the semantic diversity among constituent sentences increases, the pooled representation necessarily shifts away from every individual sentence embedding, yielding a smoothed and less discriminative vector. Building on this foundation, we formalize semantic shift as a computable measure integrating local semantic evolution and global semantic dispersion. Through controlled experiments across corpora and multiple embedding models, we show that semantic shift aligns closely with the severity of embedding concentration and predicts retrieval degradation, whereas text length alone does not. Overall, semantic shift offers a unified and actionable lens for understanding embedding collapse and for diagnosing when anisotropy becomes harmful.