Memory-Efficient Community Detection on Large Graphs Using Weighted Sketches

TL;DR

This paper presents memory-efficient community detection algorithms for large graphs using weighted Misra-Gries sketches, reducing memory usage with minimal impact on detection quality and moderate runtime increase.

Contribution

It introduces weighted Misra-Gries sketches as a memory-efficient alternative to per-thread hashtables in Louvain, Leiden, and LPA algorithms for community detection.

Findings

Memory usage reduced significantly with weighted MG sketches.

Detection quality drops by up to 1% with the new approach.

Moderate runtime penalties observed, suitable for parallel systems.

Abstract

Community detection in graphs identifies groups of nodes with denser connections within the groups than between them, and while existing studies often focus on optimizing detection performance, memory constraints become critical when processing large graphs on shared-memory systems. We recently proposed efficient implementations of the Louvain, Leiden, and Label Propagation Algorithms (LPA) for community detection. However, these incur significant memory overhead from the use of collision-free per-thread hashtables. To address this, we introduce memory-efficient alternatives using weighted Misra-Gries (MG) sketches, which replace the per-thread hashtables, and reduce memory demands in Louvain, Leiden, and LPA implementations - while incurring only a minor quality drop (up to 1%) and moderate runtime penalties. We believe that these approaches, though slightly slower, are well-suited for parallel processing and could outperform current memory-intensive techniques on systems with many threads.