SGPT: GPT Sentence Embeddings for Semantic Search

TL;DR

SGPT leverages large decoder transformers to generate high-quality sentence embeddings for semantic search, outperforming existing methods and enabling the use of large language models for this task.

Contribution

Introduces SGPT, a method to use decoder transformers for sentence embeddings and semantic search, achieving state-of-the-art results with significantly fewer parameters.

Findings

SGPT improves sentence embedding quality by 7% over previous best.

SGPT outperforms a 175-billion-parameter model on BEIR benchmark.

Code and models are publicly available for reproducibility.

Abstract

Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.