To MRL or not to MRL: Text Embeddings are Robust to Truncation Without Matryoshka Embeddings, Except In Heavy Truncation Scenarios

TL;DR

This paper compares the effectiveness of truncation robustness in text embeddings with and without Matryoshka Representation Learning (MRL), finding that non-MRL models are often competitive unless heavy truncation is applied.

Contribution

The study provides a direct comparison between random truncation and MRL, revealing that non-MRL models are robust to truncation and often outperform MRL models unless in heavy truncation scenarios.

Findings

Non-MRL models are competitive with MRL models under moderate truncation.

Heavy truncation (80% reduction) favors MRL models.

Truncation robustness may not solely depend on MRL training.

Abstract

Matryoshka Representation Learning (MRL) is a widely adopted approach for training text encoders so they provide useful text representations at various sizes, available by simply truncating the resulting vectors at sizes pre-determined at training time. Recent works have shown that randomly truncating text embeddings has minimal impact in downstream performance unless vectors are reduced in size by at least 70%, suggesting that embeddings are already robust to truncation without the use of MRL. However, no prior work has compared random truncation to MRL, so it is unclear how the two methods compare as effective embedding reduction methods. In this paper, we study this by applying the same truncation used by MRL to models trained with and without MRL. Our results across several models and downstream tasks show that, unless heavily truncating embeddings (i.e. reducing their size by at least 80%), truncated embeddings of non-MRL models are competitive with, and often outperform models trained with MRL. This suggests that truncation robustness may not necessarily come from MRL, and that the choice of spending the additional training cost of MRL depends on whether heavy truncation is desired.