Language Models are Universal Embedders

TL;DR

This paper demonstrates that pre-trained multilingual large language models can serve as effective universal embedders across various languages and tasks, even without task-specific fine-tuning.

Contribution

It introduces simple methods to create universal embedding models from multilingual LLMs and provides a benchmark to evaluate their performance across diverse scenarios.

Findings

Models produce high-quality embeddings across languages

Effective for multiple tasks without additional training

Supports languages and tasks with no finetuning data

Abstract

In the large language model (LLM) revolution, embedding is a key component of various systems, such as retrieving knowledge or memories for LLMs or building content moderation filters. As such cases span from English to other natural or programming languages, from retrieval to classification and beyond, it is advantageous to build a unified embedding model rather than dedicated ones for each scenario. In this context, the pre-trained multilingual decoder-only large language models, e.g., BLOOM, emerge as a viable backbone option. To assess their potential, we propose straightforward strategies for constructing embedders and introduce a universal evaluation benchmark. Experimental results show that our trained model is proficient at generating good embeddings across languages and tasks, even extending to languages and tasks for which no finetuning/pretraining data is available. We also present detailed analyses and additional evaluations. We hope that this work could encourage the development of more robust open-source universal embedders.