Comparing Apples and Oranges: Measuring Differences between Exploratory Data Mining Results

TL;DR

This paper introduces a novel information-theoretic measure to compare exploratory data mining results, especially on binary data, enabling meaningful assessment of differences and aiding iterative data analysis.

Contribution

It proposes a flexible, interpretable measure based on Maximum Entropy and Kullback-Leibler divergence that incorporates background knowledge and generalizes existing dissimilarity metrics.

Findings

The measure effectively distinguishes between different mining methods.

It can identify parts of results that redescribe others.

It supports iterative data mining for maximal novel insights.

Abstract

Deciding whether the results of two different mining algorithms provide significantly different information is an important, yet understudied, open problem in exploratory data mining. Whether the goal is to select the most informative result for analysis, or to decide which mining approach will most likely provide the most novel insight, it is essential that we can tell how different the information is that different results by possibly different methods provide. In this paper we take a first step towards comparing exploratory data mining results on binary data. We propose to meaningfully convert results into sets of noisy tiles, and compare between these sets by Maximum Entropy modelling and Kullback-Leibler divergence, well-founded notions from Information Theory. We so construct a measure that is highly flexible, and allows us to naturally include background knowledge, such that differences in results can be measured from the perspective of what a user already knows. Furthermore, adding to its interpretability, it coincides with Jaccard dissimilarity when we only consider exact tiles. Our approach provides a means to study and tell differences between results of different exploratory data mining methods. As an application, we show that our measure can also be used to identify which parts of results best redescribe other results. Furthermore, we study its use for iterative data mining, where one iteratively wants to find that result that will provide maximal novel information. Experimental evaluation shows our measure gives meaningful results, correctly identifies methods that are similar in nature, automatically provides sound redescriptions of results, and is highly applicable for iterative data mining.