The generalized underlap coefficient with an application in clustering

TL;DR

This paper introduces a generalized underlap coefficient (UNL) for multivariate data, establishing its properties, connections to other measures, and demonstrating its utility in clustering analysis with real datasets.

Contribution

It extends the underlap coefficient to multivariate variables, provides theoretical properties, and applies it to assess covariate dependence in clustering.

Findings

UNL effectively measures multi-group separation in multivariate data.

The importance sampling estimator enables efficient computation of UNL.

Application to real datasets demonstrates UNL's practical utility in clustering analysis.

Abstract

Quantifying distributional separation across groups is fundamental in statistical learning and scientific discovery, yet most classical discrepancy measures are tailored to two-group comparisons. We generalize the underlap coefficient (UNL), a multi-group separation measure, to multivariate variables. We establish key properties of the UNL and provide an explicit connection to total variation. We further interpret the UNL as a dependence measure between a group label and variables of interest and compare it with mutual information. We propose an efficient importance sampling estimator of the UNL that can be combined with flexible density estimators. The utility of the UNL for assessing partition-covariate dependence in clustering is highlighted in detail, where it is particularly useful for evaluating whether the latent group structure can be explained by specific covariates. Finally we illustrate the application of the UNL in clustering using two real world datasets.