An Empirical Comparison of Methods for Quantifying the Similarity of Categorical Datasets

TL;DR

This paper empirically compares various methods for measuring the similarity of categorical datasets, highlighting their strengths, resource use, and scenarios where each performs best.

Contribution

It provides a neutral comparison of promising similarity measures for categorical data, guiding method selection based on dataset differences and resource constraints.

Findings

Edge count tests perform well for two datasets.

Constrained minimum (CM) distance can outperform others in certain scenarios.

Friedman-Rafsky test is recommended for two-sample comparisons.

Abstract

Quantifying the similarity of two or more datasets has widespread applications in statistics and machine learning. The method choice is, however, difficult due to the abundance of proposed methods and the lack of neutral comparison studies, especially for categorical data. Here, the most promising methods are compared concerning their ability to detect certain differences between datasets and their resource consumption. The results show that the edge count tests perform well when comparing two datasets (i.e., the two-sample case). For certain scenarios, the constrained minimum (CM) distance performs even better. For categorical data consisting of variables with five categories each, the best method depends on the type of difference between the distributions, with either the CM distance and certain graph-based tests performing best, or the classifier-based tests (C2ST). This tendency is even clearer for multiple datasets. Overall, the Friedman-Rafsky test can be recommended for two samples as a compromise of high performance, acceptable resource consumption, and computational error occurrences. For the multi-sample case, the Multi-Sample Mahalanobis Cross-Match (MMCM) test can be recommended due to its comparably good performance and low resource consumption.