Subjectively Interesting Subgroup Discovery on Real-valued Targets

TL;DR

This paper presents a novel method for discovering subjectively interesting subgroups in high-dimensional data with real-valued targets, leveraging information theory and prior knowledge to efficiently identify informative, non-redundant patterns.

Contribution

It introduces a new approach that combines subjective interestingness with information theory for subgroup discovery involving real-valued attributes, supporting iterative data mining.

Findings

Effective identification of informative subgroups in real-valued data

Supports incorporation of prior knowledge for more relevant pattern discovery

Enables iterative exploration of high-dimensional datasets

Abstract

Deriving insights from high-dimensional data is one of the core problems in data mining. The difficulty mainly stems from the fact that there are exponentially many variable combinations to potentially consider, and there are infinitely many if we consider weighted combinations, even for linear combinations. Hence, an obvious question is whether we can automate the search for interesting patterns and visualizations. In this paper, we consider the setting where a user wants to learn as efficiently as possible about real-valued attributes. For example, to understand the distribution of crime rates in different geographic areas in terms of other (numerical, ordinal and/or categorical) variables that describe the areas. We introduce a method to find subgroups in the data that are maximally informative (in the formal Information Theoretic sense) with respect to a single or set of real-valued target attributes. The subgroup descriptions are in terms of a succinct set of arbitrarily-typed other attributes. The approach is based on the Subjective Interestingness framework FORSIED to enable the use of prior knowledge when finding most informative non-redundant patterns, and hence the method also supports iterative data mining.