Human-Guided Data Exploration

TL;DR

This paper introduces a Human-Guided Data Exploration framework that enhances interactive data analysis by allowing users to incorporate prior knowledge, focus on specific data subsets, and compare complex hypotheses, supported by an open-source tool.

Contribution

It generalizes previous iterative data mining methods by enabling user steering, subset focus, and hypothesis comparison, using an efficient constrained randomisation scheme.

Findings

User-guided exploration improves focus on relevant data subsets.

Hypothesis comparison aids in understanding complex data relations.

Framework is effective on real-world datasets.

Abstract

The outcome of the explorative data analysis (EDA) phase is vital for successful data analysis. EDA is more effective when the user interacts with the system used to carry out the exploration. In the recently proposed paradigm of iterative data mining the user controls the exploration by inputting knowledge in the form of patterns observed during the process. The system then shows the user views of the data that are maximally informative given the user's current knowledge. Although this scheme is good at showing surprising views of the data to the user, there is a clear shortcoming: the user cannot steer the process. In many real cases we want to focus on investigating specific questions concerning the data. This paper presents the Human Guided Data Exploration framework, generalising previous research. This framework allows the user to incorporate existing knowledge into the exploration process, focus on exploring a subset of the data, and compare different complex hypotheses concerning relations in the data. The framework utilises a computationally efficient constrained randomisation scheme. To showcase the framework, we developed a free open-source tool, using which the empirical evaluation on real-world datasets was carried out. Our evaluation shows that the ability to focus on particular subsets and being able to compare hypotheses are important additions to the interactive iterative data mining process.