LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States

TL;DR

This paper shows that attention value vectors in LLMs better capture sentence semantics than hidden states, introducing a simple, training-free method called Value Aggregation that outperforms existing embeddings.

Contribution

It introduces Value Aggregation (VA) and AlignedWVA, novel methods leveraging attention values for improved sentence embeddings without training, surpassing prior approaches.

Findings

VA outperforms other LLM-based embeddings in a training-free setting

AlignedWVA achieves state-of-the-art results among training-free embeddings

Fine-tuning VA can further enhance LLM embedding quality

Abstract

Sentence representations are foundational to many Natural Language Processing (NLP) applications. While recent methods leverage Large Language Models (LLMs) to derive sentence representations, most rely on final-layer hidden states, which are optimized for next-token prediction and thus often fail to capture global, sentence-level semantics. This paper introduces a novel perspective, demonstrating that attention value vectors capture sentence semantics more effectively than hidden states. We propose Value Aggregation (VA), a simple method that pools token values across multiple layers and token indices. In a training-free setting, VA outperforms other LLM-based embeddings, even matches or surpasses the ensemble-based MetaEOL. Furthermore, we demonstrate that when paired with suitable prompts, the layer attention outputs can be interpreted as aligned weighted value vectors. Specifically, the attention scores of the last token function as the weights, while the output projection matrix ($W_O$) aligns these weighted value vectors with the common space of the LLM residual stream. This refined method, termed Aligned Weighted VA (AlignedWVA), achieves state-of-the-art performance among training-free LLM-based embeddings, outperforming the high-cost MetaEOL by a substantial margin. Finally, we highlight the potential of obtaining strong LLM embedding models through fine-tuning Value Aggregation.