Approximating Spectral Clustering via Sampling: a Review

TL;DR

This review examines sampling-based spectral clustering methods that aim to reduce computational costs, analyzing their theoretical guarantees, practical benefits, and limitations, and highlighting the gap between theory and real-world application.

Contribution

It provides a comprehensive overview of sampling-based spectral clustering techniques with theoretical performance guarantees, discussing their practical effectiveness and current limitations.

Findings

Most scalable methods are intuitively motivated or loosely controlled.

End-to-end guarantees often rely on strong assumptions.

There is a gap between theoretical guarantees and practical performance.

Abstract

Spectral clustering refers to a family of unsupervised learning algorithms that compute a spectral embedding of the original data based on the eigenvectors of a similarity graph. This non-linear transformation of the data is both the key of these algorithms' success and their Achilles heel: forming a graph and computing its dominant eigenvectors can indeed be computationally prohibitive when dealing with more that a few tens of thousands of points. In this paper, we review the principal research efforts aiming to reduce this computational cost. We focus on methods that come with a theoretical control on the clustering performance and incorporate some form of sampling in their operation. Such methods abound in the machine learning, numerical linear algebra, and graph signal processing literature and, amongst others, include Nystr\"om-approximation, landmarks, coarsening, coresets, and compressive spectral clustering. We present the approximation guarantees available for each and discuss practical merits and limitations. Surprisingly, despite the breadth of the literature explored, we conclude that there is still a gap between theory and practice: the most scalable methods are only intuitively motivated or loosely controlled, whereas those that come with end-to-end guarantees rely on strong assumptions or enable a limited gain of computation time.