KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs

TL;DR

KV-Embedding is a training-free method that re-routes internal key-value states in decoder-only LLMs to produce better text embeddings by accessing sequence-level context within a single forward pass.

Contribution

The paper introduces KV-Embedding, a novel approach that activates frozen LLMs' internal states for improved embeddings without additional training.

Findings

Outperforms existing training-free baselines by up to 10%

Maintains robust performance on sequences up to 4,096 tokens

Demonstrates effectiveness across multiple LLM backbones

Abstract

While LLMs are powerful embedding backbones, their application in training-free settings faces two structural challenges: causal attention restricts early tokens from accessing subsequent context, and the next-token prediction objective biases representations toward generation rather than semantic compression. To address these limitations, we propose KV-Embedding, a framework that activates the latent representation power of frozen LLMs. Our method leverages the observation that the key-value (KV) states of the final token at each layer encode a compressed view of the sequence. By re-routing these states as a prepended prefix, we enable all tokens to access sequence-level context within a single forward pass. To ensure model-agnostic applicability, we introduce an automated layer selection strategy based on intrinsic dimensionality. Evaluations on MTEB across Qwen, Mistral, and Llama backbones show that KV-Embedding outperforms existing training-free baselines by up to 10%, while maintaining robust performance on sequences up to 4,096 tokens. These results demonstrate that internal state manipulation offers an efficient alternative to input modification, and we hope this work encourages further exploration of LLM internals for representation learning.