An Empirical Comparison of Methods for Quantifying the Similarity of Numeric Datasets

TL;DR

This paper empirically compares 36 methods for measuring dataset similarity across various scenarios, providing a ranking and decision rules to select the most effective methods for different data characteristics.

Contribution

It offers a comprehensive evaluation and ranking of similarity measures, along with recommended combinations for different dataset comparison scenarios.

Findings

Identifies top-performing methods for dataset similarity detection.

Provides decision rules for choosing methods based on dataset characteristics.

Proposes method combinations that perform near-optimally in most scenarios.

Abstract

Methods for quantifying the similarity of datasets are relevant in applications where two or more datasets, or their underlying distributions, need to be compared, ranging from two- and k-sample testing to applications in machine learning and synthetic data generation. Many methods for quantifying the similarity of datasets are available from the literature, but due to the lack of neutral comparison studies, it is unclear which method to choose when. Here, 36 methods applicable to continuous data are compared across various scenarios, including two or more datasets drawn from different distributions. Several deviations between datasets are considered, including shift and scale alternatives or differences in higher moments. An overall method ranking is established based on the methods' abilities to differentiate between datasets from different distributions, combined with computational aspects. Based on this, concrete decision rules for finding the best method based on characteristics of the datasets are determined. Moreover, combinations of four to six methods are proposed in the two-sample case such that in 90% to 95% of the considered scenarios, at least one of these methods is almost as good as the best method. In the multi-sample case, a combination of two to three methods is proposed analogously.