Generalization Performance of Ensemble Clustering: From Theory to Algorithm

TL;DR

This paper provides a theoretical analysis of ensemble clustering's generalization performance, deriving bounds and conditions for consistency, and introduces a new algorithm that improves clustering accuracy based on these insights.

Contribution

It establishes theoretical bounds for ensemble clustering's generalization error, proves conditions for consistency, and develops a novel algorithm that enhances performance by balancing bias and diversity.

Findings

Derived convergence and excess risk bounds of order $\\mathcal{O}(\rac{\log n}{m})$

Proved ensemble clustering is consistent when $m,n \to \infty$ and $m \gg \log n$

Achieved average improvements of 6.1%, 7.3%, and 6.0% on 10 datasets for NMI, ARI, and Purity

Abstract

Ensemble clustering has demonstrated great success in practice; however, its theoretical foundations remain underexplored. This paper examines the generalization performance of ensemble clustering, focusing on generalization error, excess risk and consistency. We derive a convergence rate of generalization error bound and excess risk bound both of $\mathcal{O}(\sqrt{\frac{\log n}{m}}+\frac{1}{\sqrt{n}})$, with $n$ and $m$ being the numbers of samples and base clusterings. Based on this, we prove that when $m$ and $n$ approach infinity and $m$ is significantly larger than log $n$, i.e., $m,n\to \infty, m\gg \log n$, ensemble clustering is consistent. Furthermore, recognizing that $n$ and $m$ are finite in practice, the generalization error cannot be reduced to zero. Thus, by assigning varying weights to finite clusterings, we minimize the error between the empirical average clusterings and their expectation. From this, we theoretically demonstrate that to achieve better clustering performance, we should minimize the deviation (bias) of base clustering from its expectation and maximize the differences (diversity) among various base clusterings. Additionally, we derive that maximizing diversity is nearly equivalent to a robust (min-max) optimization model. Finally, we instantiate our theory to develop a new ensemble clustering algorithm. Compared with SOTA methods, our approach achieves average improvements of 6.1%, 7.3%, and 6.0% on 10 datasets w.r.t. NMI, ARI, and Purity. The code is available at https://github.com/xuz2019/GPEC.