Sticking to the Mean: Detecting Sticky Tokens in Text Embedding Models

TL;DR

This paper identifies and analyzes 'sticky tokens' in Transformer-based text embedding models, introduces an efficient detection method, and demonstrates their negative impact on downstream NLP task performance.

Contribution

It formally defines sticky tokens, presents the Sticky Token Detector (STD), and provides extensive analysis across multiple models revealing their origins and effects.

Findings

868 sticky tokens identified across 14 models

Sticky tokens cause up to 50% performance degradation

Sticky tokens often originate from special or fragmented subwords

Abstract

Despite the widespread use of Transformer-based text embedding models in NLP tasks, surprising 'sticky tokens' can undermine the reliability of embeddings. These tokens, when repeatedly inserted into sentences, pull sentence similarity toward a certain value, disrupting the normal distribution of embedding distances and degrading downstream performance. In this paper, we systematically investigate such anomalous tokens, formally defining them and introducing an efficient detection method, Sticky Token Detector (STD), based on sentence and token filtering. Applying STD to 40 checkpoints across 14 model families, we discover a total of 868 sticky tokens. Our analysis reveals that these tokens often originate from special or unused entries in the vocabulary, as well as fragmented subwords from multilingual corpora. Notably, their presence does not strictly correlate with model size or vocabulary size. We further evaluate how sticky tokens affect downstream tasks like clustering and retrieval, observing significant performance drops of up to 50%. Through attention-layer analysis, we show that sticky tokens disproportionately dominate the model's internal representations, raising concerns about tokenization robustness. Our findings show the need for better tokenization strategies and model design to mitigate the impact of sticky tokens in future text embedding applications.