Preconditioned Spectral Clustering for Stochastic Block Partition Streaming Graph Challenge

TL;DR

This paper presents an efficient spectral clustering method using preconditioned eigenvalue solvers for static and streaming graphs, significantly reducing computation time while maintaining high accuracy in identifying graph partitions.

Contribution

The paper introduces a preconditioned spectral clustering approach with LOBPCG that accelerates eigenvalue computations for static and streaming graphs, outperforming baseline methods in speed and accuracy.

Findings

Achieves 10x error reduction in 10-20 iterations without preconditioning.

Clusters 98-160 groups in seconds for large graphs, outperforming baseline times.

Streaming approach reduces iterations and maintains cluster quality over multiple stages.

Abstract

Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) is demonstrated to efficiently solve eigenvalue problems for graph Laplacians that appear in spectral clustering. For static graph partitioning, 10-20 iterations of LOBPCG without preconditioning result in ~10x error reduction, enough to achieve 100% correctness for all Challenge datasets with known truth partitions, e.g., for graphs with 5K/.1M (50K/1M) Vertices/Edges in 2 (7) seconds, compared to over 5,000 (30,000) seconds needed by the baseline Python code. Our Python code 100% correctly determines 98 (160) clusters from the Challenge static graphs with 0.5M (2M) vertices in 270 (1,700) seconds using 10GB (50GB) of memory. Our single-precision MATLAB code calculates the same clusters at half time and memory. For streaming graph partitioning, LOBPCG is initiated with approximate eigenvectors of the graph Laplacian already computed for the previous graph, in many cases reducing 2-3 times the number of required LOBPCG iterations, compared to the static case. Our spectral clustering is generic, i.e. assuming nothing specific of the block model or streaming, used to generate the graphs for the Challenge, in contrast to the base code. Nevertheless, in 10-stage streaming comparison with the base code for the 5K graph, the quality of our clusters is similar or better starting at stage 4 (7) for emerging edging (snowballing) streaming, while the computations are over 100-1000 faster.