Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources

TL;DR

Source2Synth is a scalable method for generating high-quality synthetic data grounded in real data sources, improving large language model performance on reasoning tasks by filtering low-quality generations.

Contribution

It introduces a novel approach that produces and curates synthetic data with reasoning steps, enhancing dataset quality for complex question answering tasks.

Findings

Improves TQA accuracy by 25.51% on WikiSQL.

Enhances MHQA performance by 22.57% on HotpotQA.

Effectively filters low-quality synthetic data.

Abstract

Synthetic data generation has recently emerged as a promising approach for enhancing the capabilities of large language models (LLMs) without the need for expensive human annotations. However, existing methods often generate data that can be low quality or contrived. In this paper, we introduce Source2Synth, a scalable approach for synthetic data generation and curation that is grounded in real-world data sources. Source2Synth takes as input a custom data source and produces synthetic data examples with intermediate reasoning steps. Our method improves the dataset quality by discarding low-quality generations based on their answerability. We demonstrate the generality of this approach by applying it to two tasks that leverage two different types of data: multi-hop question answering (MHQA), where we test complex reasoning abilities leveraging documents, and tabular question answering (TQA), where we test tool usage leveraging tables. Our method improves performance by 25.51% for TQA on WikiSQL and 22.57% for MHQA on HotpotQA compared to the fine-tuned baselines.