Improving Sentence Embeddings with Automatic Generation of Training Data Using Few-shot Examples

TL;DR

This paper proposes a method to improve sentence embeddings by automatically generating training data using few-shot learning with large language models, eliminating the need for large annotated datasets.

Contribution

It introduces a novel approach to generate training data automatically for sentence embedding models using few-shot examples, enhancing performance without manual annotations.

Findings

Outperforms existing models on semantic textual similarity tasks

Effective automatic dataset generation with few-shot learning

Reduces dependency on manually annotated datasets

Abstract

Decoder-based large language models (LLMs) have shown high performance on many tasks in natural language processing. This is also true for sentence embedding learning, where a decoder-based model, PromptEOL, has achieved the best performance on semantic textual similarity (STS) tasks. However, PromptEOL requires a manually annotated natural language inference (NLI) dataset for fine-tuning. We aim to improve sentence embeddings without using large manually annotated datasets by automatically generating an NLI dataset with an LLM and using it for fine-tuning of PromptEOL. To achieve this, we explore methods of data generation suitable for sentence embedding learning in this study. Specifically, we will focus on automatic dataset generation through few-shot learning and explore the appropriate methods to leverage few-shot examples. Experimental results on the STS tasks demonstrate that our approach outperforms existing models in settings without large manually annotated datasets.