WoCE: a framework for clustering ensemble by exploiting the wisdom of Crowds theory

TL;DR

This paper introduces WoCE, a novel clustering ensemble framework inspired by the Wisdom of Crowds theory, which improves clustering accuracy by leveraging diversity, independence, decentralization, and aggregation conditions.

Contribution

The paper proposes a new clustering ensemble method that incorporates WOC principles, including a novel independence criterion and a weighted evidence accumulation approach.

Findings

Outperforms state-of-the-art clustering ensemble methods

Demonstrates effectiveness across various datasets

Introduces novel metrics and mechanisms for clustering quality

Abstract

The Wisdom of Crowds (WOC), as a theory in the social science, gets a new paradigm in computer science. The WOC theory explains that the aggregate decision made by a group is often better than those of its individual members if specific conditions are satisfied. This paper presents a novel framework for unsupervised and semi-supervised cluster ensemble by exploiting the WOC theory. We employ four conditions in the WOC theory, i.e., diversity, independency, decentralization and aggregation, to guide both the constructing of individual clustering results and the final combination for clustering ensemble. Firstly, independency criterion, as a novel mapping system on the raw data set, removes the correlation between features on our proposed method. Then, decentralization as a novel mechanism generates high-quality individual clustering results. Next, uniformity as a new diversity metric evaluates the generated clustering results. Further, weighted evidence accumulation clustering method is proposed for the final aggregation without using thresholding procedure. Experimental study on varied data sets demonstrates that the proposed approach achieves superior performance to state-of-the-art methods.