Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders

TL;DR

EPIC introduces an embedding-based in-context prompt training method that improves LLM-generated embeddings, reduces computational costs, and achieves state-of-the-art results on the MTEB benchmark.

Contribution

The paper presents a novel embedding-based prompt training strategy that enhances embedding quality and efficiency, surpassing existing models on key benchmarks.

Findings

EPIC achieves state-of-the-art results on the MTEB benchmark.

The method reduces token overhead compared to traditional in-context learning.

EPIC-trained models perform well with or without in-context prompts.

Abstract

Large language models (LLMs) have been widely explored for embedding generation. While recent studies show that in-context learning (ICL) effectively enhances the representational capability of LLMs by prepending a few task-related demonstrations, it causes substantial token overhead due to the increased sequence length. In this work, we propose EPIC, a novel embedding-based in-context prompt training strategy that leverages ICL to generate high-quality embeddings while reducing computational burden during both training and inference. This approach replaces discrete text demonstrations with their corresponding continuous embeddings, which not only encourages the LLM to align semantically-related text pairs during contrastive learning, but also requires the model to interpret demonstration embeddings as part of the in-context prompt. Consequently, EPIC-trained models achieve excellent embedding performance both with or without in-context prompts at inference time. Comprehensive experiments demonstrate that our method establishes new state-of-the-art results on the MTEB benchmark, surpassing frontier models trained solely on publicly available retrieval data. Extensive ablation studies further validate the effectiveness and necessity of our mechanism.