Indexing the Earth Mover's Distance Using Normal Distributions

TL;DR

This paper introduces a novel indexing method using normal distribution approximations and Hough space transformations to accelerate Earth Mover's Distance based K-NN queries on uncertain data sets.

Contribution

It proposes a new lower bound and an index structure that significantly improves EMD query performance for uncertain databases.

Findings

Reduces K-NN query time on uncertain data

Scales well with large and complex datasets

Effective for heterogeneous uncertain data sets

Abstract

Querying uncertain data sets (represented as probability distributions) presents many challenges due to the large amount of data involved and the difficulties comparing uncertainty between distributions. The Earth Mover's Distance (EMD) has increasingly been employed to compare uncertain data due to its ability to effectively capture the differences between two distributions. Computing the EMD entails finding a solution to the transportation problem, which is computationally intensive. In this paper, we propose a new lower bound to the EMD and an index structure to significantly improve the performance of EMD based K-nearest neighbor (K-NN) queries on uncertain databases. We propose a new lower bound to the EMD that approximates the EMD on a projection vector. Each distribution is projected onto a vector and approximated by a normal distribution, as well as an accompanying error term. We then represent each normal as a point in a Hough transformed space. We then use the concept of stochastic dominance to implement an efficient index structure in the transformed space. We show that our method significantly decreases K-NN query time on uncertain databases. The index structure also scales well with database cardinality. It is well suited for heterogeneous data sets, helping to keep EMD based queries tractable as uncertain data sets become larger and more complex.