When Text Embedding Meets Large Language Model: A Comprehensive Survey

TL;DR

This survey comprehensively reviews how large language models (LLMs) are integrated with text embeddings, categorizing recent research into three themes and discussing future challenges and directions in NLP.

Contribution

It provides a systematic overview of the interplay between LLMs and text embeddings, organizing recent works by interaction patterns and highlighting unresolved challenges.

Findings

Categorizes LLM and text embedding interactions into three themes

Highlights unresolved challenges from pre-LLM and LLM eras

Outlines future research directions in text embedding and LLM integration

Abstract

Text embedding has become a foundational technology in natural language processing (NLP) during the deep learning era, driving advancements across a wide array of downstream tasks. While many natural language understanding challenges can now be modeled using generative paradigms and leverage the robust generative and comprehension capabilities of large language models (LLMs), numerous practical applications - such as semantic matching, clustering, and information retrieval - continue to rely on text embeddings for their efficiency and effectiveness. Therefore, integrating LLMs with text embeddings has become a major research focus in recent years. In this survey, we categorize the interplay between LLMs and text embeddings into three overarching themes: (1) LLM-augmented text embedding, enhancing traditional embedding methods with LLMs; (2) LLMs as text embedders, adapting their innate capabilities for high-quality embedding; and (3) Text embedding understanding with LLMs, leveraging LLMs to analyze and interpret embeddings. By organizing recent works based on interaction patterns rather than specific downstream applications, we offer a novel and systematic overview of contributions from various research and application domains in the era of LLMs. Furthermore, we highlight the unresolved challenges that persisted in the pre-LLM era with pre-trained language models (PLMs) and explore the emerging obstacles brought forth by LLMs. Building on this analysis, we outline prospective directions for the evolution of text embedding, addressing both theoretical and practical opportunities in the rapidly advancing landscape of NLP.