Clusterability test for categorical data

TL;DR

This paper introduces TestCat, a statistically rigorous method for evaluating the clusterability of categorical data by analyzing attribute associations, addressing a previously overlooked problem in cluster analysis.

Contribution

The paper presents the first effective statistical test for categorical data clusterability, using chi-squared statistics to assess attribute associations.

Findings

TestCat outperforms existing numeric data clusterability methods on benchmark datasets.

It provides an analytical p-value to determine clusterability of categorical data.

The approach is the first to reliably evaluate categorical data clusterability.

Abstract

The objective of clusterability evaluation is to check whether a clustering structure exists within the data set. As a crucial yet often-overlooked issue in cluster analysis, it is essential to conduct such a test before applying any clustering algorithm. If a data set is unclusterable, any subsequent clustering analysis would not yield valid results. Despite its importance, the majority of existing studies focus on numerical data, leaving the clusterability evaluation issue for categorical data as an open problem. Here we present TestCat, a testing-based approach to assess the clusterability of categorical data in terms of an analytical $p$-value. The key idea underlying TestCat is that clusterable categorical data possess many strongly associated attribute pairs and hence the sum of chi-squared statistics of all attribute pairs is employed as the test statistic for $p$-value calculation. We apply our method to a set of benchmark categorical data sets, showing that TestCat outperforms those solutions based on existing clusterability evaluation methods for numeric data. To the best of our knowledge, our work provides the first way to effectively recognize the clusterability of categorical data in a statistically sound manner.