Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe

TL;DR

This paper presents an algorithm to optimize the training configuration of text embedding models derived from pre-trained language models, balancing model size, data, and fine-tuning methods for different compute budgets.

Contribution

The authors introduce a compute-optimal recipe for training text embedding models by systematically exploring configurations of model size, data, and fine-tuning techniques.

Findings

Full fine-tuning is optimal at lower computational budgets.

Low-rank adaptation fine-tuning is better at higher budgets.

The proposed recipe guides practitioners in designing efficient embedding models.

Abstract

Text embeddings are essential for many tasks, such as document retrieval, clustering, and semantic similarity assessment. In this paper, we study how to contrastively train text embedding models in a compute-optimal fashion, given a suite of pre-trained decoder-only language models. Our innovation is an algorithm that produces optimal configurations of model sizes, data quantities, and fine-tuning methods for text-embedding models at different computational budget levels. The resulting recipe, which we obtain through extensive experiments, can be used by practitioners to make informed design choices for their embedding models. Specifically, our findings suggest that full fine-tuning and low-rank adaptation fine-tuning produce optimal models at lower and higher computational budgets respectively.