A Deterministic Information Bottleneck Method for Clustering Mixed-Type Data

TL;DR

This paper introduces DIBmix, a novel information-theoretic clustering method for mixed continuous and categorical data, which outperforms existing algorithms especially with imbalanced clusters and moderate overlaps.

Contribution

It extends the Information Bottleneck principle to heterogeneous data using generalized kernels, with a systematic bandwidth selection and adaptive hyperparameter scheme.

Findings

DIBmix outperforms four established clustering methods in various benchmarks.

The method is particularly effective with imbalanced clusters and moderate overlaps.

DIBmix is a theoretically grounded alternative to traditional centroid-based clustering algorithms.

Abstract

In this paper, we present an information-theoretic method for clustering mixed-type data, that is, data consisting of both continuous and categorical variables. The proposed approach extends the Information Bottleneck principle to heterogeneous data through generalised product kernels, integrating continuous, nominal, and ordinal variables within a unified optimization framework. We address the following challenges: developing a systematic bandwidth selection strategy that equalises contributions across variable types, and proposing an adaptive hyperparameter updating scheme that ensures a valid solution into a predetermined number of potentially imbalanced clusters. Through simulations on 28,800 synthetic data sets and ten publicly available benchmarks, we demonstrate that the proposed method, named DIBmix, achieves superior performance compared to four established methods (KAMILA, K-Prototypes, FAMD with K-Means, and PAM with Gower's dissimilarity). Results show DIBmix particularly excels when clusters exhibit size imbalances, data contain low or moderate cluster overlap, and categorical and continuous variables are equally represented. The method presents a significant advantage over traditional centroid-based algorithms, establishing DIBmix as a competitive and theoretically grounded alternative for mixed-type data clustering.