Computational Feasibility of Clustering under Clusterability Assumptions

TL;DR

This paper surveys recent research on the computational feasibility of clustering under clusterability assumptions, critically evaluating whether such assumptions can explain the practical efficiency of clustering algorithms.

Contribution

It provides a comprehensive review and critical analysis of recent work on clusterability notions, assessing their potential to justify efficient clustering in practice.

Findings

The CDNM thesis remains unproven and lacks formal support.

Existing notions of clusterability often do not meet necessary formal requirements.

Open research challenges include defining stronger clusterability conditions.

Abstract

It is well known that most of the common clustering objectives are NP-hard to optimize. In practice, however, clustering is being routinely carried out. One approach for providing theoretical understanding of this seeming discrepancy is to come up with notions of clusterability that distinguish realistically interesting input data from worst-case data sets. The hope is that there will be clustering algorithms that are provably efficient on such 'clusterable' instances. In other words, hope that "Clustering is difficult only when it does not matter" (CDNM thesis, for short). We believe that to some extent this may indeed be the case. This paper provides a survey of recent papers along this line of research and a critical evaluation their results. Our bottom line conclusion is that that CDNM thesis is still far from being formally substantiated. We start by discussing which requirements should be met in order to provide formal support the validity of the CDNM thesis. In particular, we list some implied requirements for notions of clusterability. We then examine existing results in view of those requirements and outline some research challenges and open questions.