AIDE: An Automated Sample-based Approach for Interactive Data Exploration

TL;DR

AIDE is an automated framework that guides users through complex datasets by intelligently sampling and predicting data patterns, reducing effort and wait time in data exploration tasks.

Contribution

The paper introduces AIDE, a novel automated data exploration system that combines classification and optimization to efficiently discover data patterns with minimal user input.

Findings

High accuracy in predicting common conjunctive queries

Effective prediction of complex disjunctive queries with limited samples

Interactive performance with user wait times under a few seconds

Abstract

In this paper, we argue that database systems be augmented with an automated data exploration service that methodically steers users through the data in a meaningful way. Such an automated system is crucial for deriving insights from complex datasets found in many big data applications such as scientific and healthcare applications as well as for reducing the human effort of data exploration. Towards this end, we present AIDE, an Automatic Interactive Data Exploration framework that assists users in discovering new interesting data patterns and eliminate expensive ad-hoc exploratory queries. AIDE relies on a seamless integration of classification algorithms and data management optimization techniques that collectively strive to accurately learn the user interests based on his relevance feedback on strategically collected samples. We present a number of exploration techniques as well as optimizations that minimize the number of samples presented to the user while offering interactive performance. AIDE can deliver highly accurate query predictions for very common conjunctive queries with small user effort while, given a reasonable number of samples, it can predict with high accuracy complex disjunctive queries. It provides interactive performance as it limits the user wait time per iteration of exploration to less than a few seconds.