Efficient Embedding-based Synthetic Data Generation for Complex Reasoning Tasks

TL;DR

This paper introduces an embedding-based sampling pipeline for synthetic data generation that improves data diversity and model performance in complex reasoning tasks by analyzing data distribution in embedding space.

Contribution

It presents a novel embedding-based sampling method that enhances synthetic data quality and diversity, leading to better model accuracy in reasoning tasks.

Findings

Higher neighborhood density correlates with increased prediction accuracy.

Embedding-based sampling improves data diversity and model performance.

The method outperforms baseline approaches on multiple benchmarks.

Abstract

Synthetic Data Generation (SDG), leveraging Large Language Models (LLMs), has recently been recognized and broadly adopted as an effective approach to improve the performance of smaller but more resource and compute efficient LLMs through fine-tuning. A key challenge in SDG is ensuring the quality and diversity of the generated data. In this paper, we analyze the diversity and distribution of generated data in the embedding space, and demonstrate a strong correlation between the density of examples within a specific neighborhood and the accuracy of predictions on examples drawn from that region. Building on this insight, we present a targeted pipeline for embedding-based sampling that enhances data diversity and consistently improves performance across several benchmarks.