SynQuE: Estimating Synthetic Dataset Quality Without Annotations

TL;DR

SynQuE introduces a framework to estimate the quality of synthetic datasets for real-world tasks without annotations, using proxy metrics and large language models, to improve data selection under scarcity.

Contribution

It formalizes the SynQuE problem, introduces new proxy metrics including LENS, and demonstrates their effectiveness across diverse tasks for synthetic data ranking.

Findings

LENS outperforms other proxies on complex tasks.

Synthetic data selection improves task accuracy significantly.

Proxies correlate well with real task performance.

Abstract

We introduce and formalize the Synthetic Dataset Quality Estimation (SynQuE) problem: ranking synthetic datasets by their expected real-world task performance using only limited unannotated real data. This addresses a critical and open challenge where data is scarce due to collection costs or privacy constraints. We establish the first comprehensive benchmarks for this problem by introducing and evaluating proxy metrics that choose synthetic data for training to maximize task performance on real data. We introduce the first proxy metrics for SynQuE by adapting distribution and diversity-based distance measures to our context via embedding models. To address the shortcomings of these metrics on complex planning tasks, we propose LENS, a novel proxy that leverages large language model reasoning. Our results show that SynQuE proxies correlate with real task performance across diverse tasks, including sentiment analysis, Text2SQL, web navigation, and image classification, with LENS consistently outperforming others on complex tasks by capturing nuanced characteristics. For instance, on text-to-SQL parsing, training on the top-3 synthetic datasets selected via SynQuE proxies can raise accuracy from 30.4% to 38.4 (+8.1)% on average compared to selecting data indiscriminately. This work establishes SynQuE as a practical framework for synthetic data selection under real-data scarcity and motivates future research on foundation model-based data characterization and fine-grained data selection. We release our code.