Meta-Task Prompting Elicits Embeddings from Large Language Models

TL;DR

MetaEOL introduces an unsupervised, prompt-based method for generating high-quality sentence embeddings from LLMs, outperforming contrastive models on various benchmarks without fine-tuning.

Contribution

It presents a novel meta-task prompting approach that guides LLMs to produce versatile embeddings, establishing a new scaling law for resource-efficient embedding generation.

Findings

Embeddings from MetaEOL outperform contrastive-trained models on STS benchmarks.

MetaEOL embeddings excel in downstream NLP tasks.

A new scaling law for embedding quality and resource efficiency.

Abstract

We introduce a new unsupervised text embedding method, Meta-Task Prompting with Explicit One-Word Limitation (MetaEOL), for generating high-quality sentence embeddings from Large Language Models (LLMs) without the need for model fine-tuning. Leveraging meta-task prompting, MetaEOL guides LLMs to produce embeddings through a series of carefully designed prompts that address multiple representational aspects. Our comprehensive experiments demonstrate that embeddings averaged from various meta-tasks are versatile embeddings that yield competitive performance on Semantic Textual Similarity (STS) benchmarks and excel in downstream tasks, surpassing contrastive-trained models. Our findings suggest a new scaling law, offering a versatile and resource-efficient approach for embedding generation across diverse scenarios.