Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs

TL;DR

This paper introduces Token Prepending, a training-free method that enhances sentence embeddings from large language models by allowing earlier tokens to access complete sentence information, improving performance across tasks.

Contribution

The paper proposes a novel Token Prepending technique that is simple, training-free, and can be integrated with existing prompt-based methods to improve sentence embeddings from autoregressive LLMs.

Findings

Significantly improves sentence embedding quality across various tasks.

Compatible with multiple prompt-based methods and LLMs.

Incur negligible additional inference cost.

Abstract

Extracting sentence embeddings from large language models (LLMs) is a promising direction, as LLMs have demonstrated stronger semantic understanding capabilities. Previous studies typically focus on prompt engineering to elicit sentence embeddings from LLMs by prompting the model to encode sentence information into the embedding of the last token. However, LLMs are mostly decoder-only models with causal attention and the earlier tokens in the sentence cannot attend to the latter tokens, resulting in biased encoding of sentence information and cascading effects on the final decoded token. To this end, we propose a novel Token Prepending (TP) technique that prepends each layer's decoded sentence embedding to the beginning of the sentence in the next layer's input, allowing earlier tokens to attend to the complete sentence information under the causal attention mechanism. The proposed TP technique is a plug-and-play and training-free technique, which means it can be seamlessly integrated with various prompt-based sentence embedding methods and autoregressive LLMs. Extensive experiments on various Semantic Textual Similarity (STS) tasks and downstream classification tasks demonstrate that our proposed TP technique can significantly improve the performance of existing prompt-based sentence embedding methods across different LLMs, while incurring negligible additional inference cost.