Distances between Data Sets Based on Summary Statistics

TL;DR

This paper introduces a novel distance measure between data sets based on summary statistics, which can be computed efficiently and has desirable properties, useful for data mining applications.

Contribution

It defines a new distance based on distribution sets, proves its computational efficiency, and demonstrates its applicability to binary data with real-world testing.

Findings

Distance can be computed in cubic time for general data.

For binary data, distance evaluation is linear time.

Empirical tests show the distance performs well on real data.

Abstract

The concepts of similarity and distance are crucial in data mining. We consider the problem of defining the distance between two data sets by comparing summary statistics computed from the data sets. The initial definition of our distance is based on geometrical notions of certain sets of distributions. We show that this distance can be computed in cubic time and that it has several intuitive properties. We also show that this distance is the unique Mahalanobis distance satisfying certain assumptions. We also demonstrate that if we are dealing with binary data sets, then the distance can be represented naturally by certain parity functions, and that it can be evaluated in linear time. Our empirical tests with real world data show that the distance works well.