Subgroup Discovery in Unstructured Data

TL;DR

This paper introduces a novel variational autoencoder designed to discover meaningful subgroups within unstructured, high-dimensional data like images, enhancing interpretability and subgroup quality.

Contribution

It proposes the subgroup-aware variational autoencoder, a new method that enables subgroup discovery in unstructured data where traditional attribute-based rules are ineffective.

Findings

Effective at learning high-quality subgroups

Supports interpretability of learned concepts

Outperforms existing methods in subgroup quality

Abstract

Subgroup discovery is a descriptive and exploratory data mining technique to identify subgroups in a population that exhibit interesting behavior with respect to a variable of interest. Subgroup discovery has numerous applications in knowledge discovery and hypothesis generation, yet it remains inapplicable for unstructured, high-dimensional data such as images. This is because subgroup discovery algorithms rely on defining descriptive rules based on (attribute, value) pairs, however, in unstructured data, an attribute is not well defined. Even in cases where the notion of attribute intuitively exists in the data, such as a pixel in an image, due to the high dimensionality of the data, these attributes are not informative enough to be used in a rule. In this paper, we introduce the subgroup-aware variational autoencoder, a novel variational autoencoder that learns a representation of unstructured data which leads to subgroups with higher quality. Our experimental results demonstrate the effectiveness of the method at learning subgroups with high quality while supporting the interpretability of the concepts.