Discriminative Span as a Predictor of Synthetic Data Utility via Classifier Reconstruction

TL;DR

This paper introduces a geometry-based metric in embedding space to predict the usefulness of synthetic data for improving classifier performance without needing to train models.

Contribution

It proposes a novel, model-agnostic metric that assesses synthetic data quality by measuring the span of variations relative to classifier directions.

Findings

The metric correlates strongly with downstream CNN classification accuracy.

Synthetic data capturing task-relevant variations improves classifier performance.

The approach works across multiple datasets and architectures.

Abstract

In many real-world computer vision applications, including medical imaging and industrial inspection, binary classification tasks are characterized by a severe scarcity of positive samples. A widely adopted solution is to generate synthetic positive data using image-to-image transformations applied to negative samples. However, a fundamental challenge remains: how can we reliably assess whether such synthetic data will improve downstream model performance? In this work, we propose a geometry-driven metric that predicts the utility of synthetic data without requiring model training. Our approach operates in the embedding space of a pre-trained foundation model and represents the dataset through difference vectors between samples. We evaluate whether the weight vector of a linear classifier can be expressed within the subspace spanned by these variations by measuring the relative projection error. Intuitively, if the variations induced by synthetic data capture task-relevant directions, their span can approximate the classifier, resulting in low projection error. Conversely, poor synthetic data fails to span these directions, leading to higher error. Across multiple datasets and architectures, we show that this metric exhibits strong correlation with downstream classification performance of CNNs trained on mixtures of real negative and synthetic positive data. These findings suggest that the proposed metric serves as a practical and informative tool for evaluating synthetic data quality in data-scarce settings.