Evaluating Synthetic Tabular Data Generated To Augment Small Sample Datasets

TL;DR

This paper critically evaluates methods for assessing synthetic tabular data used to augment small datasets, revealing limitations of traditional metrics and proposing a topological approach with noted instability.

Contribution

It introduces a normalized Bottleneck distance metric for evaluating synthetic data and highlights the need for multi-faceted validation strategies for small sample augmentation.

Findings

Global metrics often misrepresent true differences

Topological measures show high variability and instability

Traditional statistical tests are unreliable with small samples

Abstract

This work proposes a method to evaluate synthetic tabular data generated to augment small sample datasets. While data augmentation techniques can increase sample counts for machine learning applications, traditional validation approaches fail when applied to extremely limited sample sizes. Our experiments across four datasets reveal significant inconsistencies between global metrics and topological measures, with statistical tests producing unreliable significance values due to insufficient sample sizes. We demonstrate that common metrics like propensity scoring and MMD often suggest similarity where fundamental topological differences exist. Our proposed normalized Bottleneck distance based metric provides complementary insights but suffers from high variability across experimental runs and occasional values exceeding theoretical bounds, showing inherent instability in topological approaches for very small datasets. These findings highlight the critical need for multi-faceted evaluation methodologies when validating synthetic data generated from limited samples, as no single metric reliably captures both distributional and structural similarity.