A New Rejection Sampling Approach to $k$-$\mathtt{means}$++ With Improved Trade-Offs

TL;DR

This paper introduces a rejection sampling method to accelerate the $k$-means++ seeding algorithm, achieving faster runtimes while maintaining provable approximation guarantees, and explores a new trade-off between computational cost and solution quality.

Contribution

It proposes a rejection sampling approach for $k$-means++ that reduces runtime and offers a novel trade-off between efficiency and clustering quality.

Findings

The first method runs in $ ilde{O}( ext{nnz}( ext{X}) + eta k^2 d)$ time with $O( ext{log} k)$ guarantees.

The second method introduces a scale-invariant factor improving the trade-off between cost and quality.

Empirical results validate the theoretical improvements on real datasets.

Abstract

The $k$-$\mathtt{means}$++ seeding algorithm (Arthur & Vassilvitskii, 2007) is widely used in practice for the $k$-means clustering problem where the goal is to cluster a dataset $\mathcal{X} \subset \mathbb{R} ^d$ into $k$ clusters. The popularity of this algorithm is due to its simplicity and provable guarantee of being $O(\log k)$ competitive with the optimal solution in expectation. However, its running time is $O(|\mathcal{X}|kd)$, making it expensive for large datasets. In this work, we present a simple and effective rejection sampling based approach for speeding up $k$-$\mathtt{means}$++. Our first method runs in time $\tilde{O}(\mathtt{nnz} (\mathcal{X}) + \beta k^2d)$ while still being $O(\log k )$ competitive in expectation. Here, $\beta$ is a parameter which is the ratio of the variance of the dataset to the optimal $k$-$\mathtt{means}$ cost in expectation and $\tilde{O}$ hides logarithmic factors in $k$ and $|\mathcal{X}|$. Our second method presents a new trade-off between computational cost and solution quality. It incurs an additional scale-invariant factor of $ k^{-\Omega( m/\beta)} \operatorname{Var} (\mathcal{X})$ in addition to the $O(\log k)$ guarantee of $k$-$\mathtt{means}$++ improving upon a result of (Bachem et al, 2016a) who get an additional factor of $m^{-1}\operatorname{Var}(\mathcal{X})$ while still running in time $\tilde{O}(\mathtt{nnz}(\mathcal{X}) + mk^2d)$. We perform extensive empirical evaluations to validate our theoretical results and to show the effectiveness of our approach on real datasets.