GEM: Empowering LLM for both Embedding Generation and Language Understanding

TL;DR

This paper introduces GEM, a simple self-supervised method that enables large decoder-only LLMs to generate high-quality text embeddings without sacrificing their original generation and reasoning abilities.

Contribution

GEM is a novel approach that inserts special tokens and manipulates attention masks, allowing any decoder-only LLM to produce effective text embeddings during post-training or fine-tuning.

Findings

Significantly improves LLMs on text embedding benchmarks (MTEB).

Maintains original NLP performance on benchmarks like MMLU.

Applicable to models ranging from 1B to 8B parameters.

Abstract

Large decoder-only language models (LLMs) have achieved remarkable success in generation and reasoning tasks, where they generate text responses given instructions. However, many applications, e.g., retrieval augmented generation (RAG), still rely on separate embedding models to generate text embeddings, which can complicate the system and introduce discrepancies in understanding of the query between the embedding model and LLMs. To address this limitation, we propose a simple self-supervised approach, Generative Embedding large language Model (GEM), that enables any large decoder-only LLM to generate high-quality text embeddings while maintaining its original text generation and reasoning capabilities. Our method inserts new special token(s) into a text body, and generates summarization embedding of the text by manipulating the attention mask. This method could be easily integrated into post-training or fine tuning stages of any existing LLMs. We demonstrate the effectiveness of our approach by applying it to two popular LLM families, ranging from 1B to 8B parameters, and evaluating the transformed models on both text embedding benchmarks (MTEB) and NLP benchmarks (MMLU). The results show that our proposed method significantly improves the original LLMs on MTEB while having a minimal impact on MMLU. Our strong results indicate that our approach can empower LLMs with state-of-the-art text embedding capabilities while maintaining their original NLP performance