Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks

TL;DR

Removing up to 50% of dimensions from text embeddings minimally impacts downstream retrieval and classification performance across various models and tasks, revealing that many dimensions are redundant or even detrimental.

Contribution

This study demonstrates that significant random truncation of text embeddings has little effect on performance and challenges prior assumptions about embedding space utilization.

Findings

Removing dimensions causes less than 10% performance drop

Many dimensions are redundant or negatively impact performance

The phenomenon extends to large language model embeddings

Abstract

In this paper, we study the surprising impact that truncating text embeddings has on downstream performance. We consistently observe across 6 state-of-the-art text encoders and 26 downstream tasks, that randomly removing up to 50% of embedding dimensions results in only a minor drop in performance, less than 10%, in retrieval and classification tasks. Given the benefits of using smaller-sized embeddings, as well as the potential insights about text encoding, we study this phenomenon and find that, contrary to what is suggested in prior work, this is not the result of an ineffective use of representation space. Instead, we find that a large number of uniformly distributed dimensions actually cause an increase in performance when removed. This would explain why, on average, removing a large number of embedding dimensions results in a marginal drop in performance. We make similar observations when truncating the embeddings used by large language models to make next-token predictions on generative tasks, suggesting that this phenomenon is not isolated to classification or retrieval tasks.