One Model Is Enough: Native Retrieval Embeddings from LLM Agent Hidden States

TL;DR

This paper introduces a method for LLM agents to generate retrieval embeddings directly from their hidden states, reducing complexity and latency while maintaining high retrieval quality.

Contribution

It proposes a lightweight projection head trained with multiple loss functions to enable LLMs to produce native retrieval embeddings, eliminating the need for separate embedding models.

Findings

Retains 97% of baseline retrieval quality

Achieves competitive Recall@10 and MRR@10 on QReCC benchmark

Systematic ablations confirm effectiveness of each loss component

Abstract

LLM agents that retrieve external knowledge typically generate a search query as text, then run a separate embedding model to encode it into a vector. This two-model pipeline adds infrastructure complexity and latency, yet is redundant: the LLM already encodes the full conversational context in its hidden states. We propose equipping LLM agents with native retrieval capability by adding a lightweight projection head that maps hidden states directly into the embedding space, eliminating the need for a separate embedding model. Trained with a combination of alignment, contrastive, and rank distillation losses, our method retains 97\% of baseline retrieval quality while enabling the LLM agent to search with its own representations. Experiments on the QReCC conversational search benchmark show competitive Recall@10 and MRR@10 compared to the standard generate-then-encode pipeline, with systematic ablations confirming the contribution of each loss component.