A Flexible Iterative Framework for Consensus Clustering

TL;DR

This paper introduces a flexible iterative consensus clustering framework that determines the optimal number of clusters and improves clustering accuracy by combining multiple algorithms and refining the consensus matrix, especially effective for noisy or high-dimensional data.

Contribution

The paper presents a novel iterative consensus clustering framework that integrates multiple algorithms and uses Markov chain theory to accurately determine the number of clusters.

Findings

Successfully determines the number of clusters in various datasets.

Achieves higher accuracy than individual algorithms.

Effective for noisy and high-dimensional data.

Abstract

A novel framework for consensus clustering is presented which has the ability to determine both the number of clusters and a final solution using multiple algorithms. A consensus similarity matrix is formed from an ensemble using multiple algorithms and several values for k. A variety of dimension reduction techniques and clustering algorithms are considered for analysis. For noisy or high-dimensional data, an iterative technique is presented to refine this consensus matrix in way that encourages algorithms to agree upon a common solution. We utilize the theory of nearly uncoupled Markov chains to determine the number, k , of clusters in a dataset by considering a random walk on the graph defined by the consensus matrix. The eigenvalues of the associated transition probability matrix are used to determine the number of clusters. This method succeeds at determining the number of clusters in many datasets where previous methods fail. On every considered dataset, our consensus method provides a final result with accuracy well above the average of the individual algorithms.