Scalable Co-Clustering for Large-Scale Data through Dynamic Partitioning and Hierarchical Merging

TL;DR

This paper introduces a scalable co-clustering approach for large datasets using dynamic partitioning and hierarchical merging, significantly reducing computation time while uncovering detailed data patterns.

Contribution

It presents a novel large matrix partitioning and hierarchical merging method that enhances scalability and robustness of co-clustering for high-dimensional data.

Findings

83% reduction in computation time for dense matrices

30% reduction in computation time for sparse matrices

Effective uncovering of intricate data patterns

Abstract

Co-clustering simultaneously clusters rows and columns, revealing more fine-grained groups. However, existing co-clustering methods suffer from poor scalability and cannot handle large-scale data. This paper presents a novel and scalable co-clustering method designed to uncover intricate patterns in high-dimensional, large-scale datasets. Specifically, we first propose a large matrix partitioning algorithm that partitions a large matrix into smaller submatrices, enabling parallel co-clustering. This method employs a probabilistic model to optimize the configuration of submatrices, balancing the computational efficiency and depth of analysis. Additionally, we propose a hierarchical co-cluster merging algorithm that efficiently identifies and merges co-clusters from these submatrices, enhancing the robustness and reliability of the process. Extensive evaluations validate the effectiveness and efficiency of our method. Experimental results demonstrate a significant reduction in computation time, with an approximate 83% decrease for dense matrices and up to 30% for sparse matrices.