Testing for Unobserved Heterogeneity via k-means Clustering

TL;DR

This paper introduces a formal statistical test for determining whether data is better represented by multiple clusters rather than a single homogeneous group, applicable across various contexts and data types.

Contribution

It proposes a simple, valid testing procedure for unobserved heterogeneity using k-means clustering, effective under mild conditions and applicable in diverse scenarios.

Findings

Test has good size control in finite samples.

Effective in clustering vehicle manufacturers.

Applicable to clustering on parameters beyond the mean.

Abstract

Clustering methods such as k-means have found widespread use in a variety of applications. This paper proposes a formal testing procedure to determine whether a null hypothesis of a single cluster, indicating homogeneity of the data, can be rejected in favor of multiple clusters. The test is simple to implement, valid under relatively mild conditions (including non-normality, and heterogeneity of the data in aspects beyond those in the clustering analysis), and applicable in a range of contexts (including clustering when the time series dimension is small, or clustering on parameters other than the mean). We verify that the test has good size control in finite samples, and we illustrate the test in applications to clustering vehicle manufacturers and U.S. mutual funds.