Using Synthetic Data to estimate the True Error is theoretically and practically doable

TL;DR

This paper explores the use of high-quality synthetic data to accurately estimate a machine learning model's true error when limited labeled data is available, supported by new theoretical bounds and practical methods.

Contribution

It introduces novel generalization bounds incorporating synthetic data and proposes a method to generate optimized synthetic samples for reliable model evaluation.

Findings

Synthetic data can effectively estimate true model error.

The proposed method outperforms existing baselines in accuracy.

Theoretical bounds highlight the importance of generator quality.

Abstract

Accurately evaluating model performance is crucial for deploying machine learning systems in real-world applications. Traditional methods often require a sufficiently large labeled test set to ensure a reliable evaluation. However, in many contexts, a large labeled dataset is costly and labor-intensive. Therefore, we sometimes have to do evaluation by a few labeled samples, which is theoretically challenging. Recent advances in generative models offer a promising alternative by enabling the synthesis of high-quality data. In this work, we make a systematic investigation about the use of synthetic data to estimate the test error of a trained model under limited labeled data conditions. To this end, we develop novel generalization bounds that take synthetic data into account. Those bounds suggest novel ways to optimize synthetic samples for evaluation and theoretically reveal the significant role of the generator's quality. Inspired by those bounds, we propose a theoretically grounded method to generate optimized synthetic data for model evaluation. Experimental results on simulation and tabular datasets demonstrate that, compared to existing baselines, our method achieves accurate and more reliable estimates of the test error.