LocalKMeans: Convergence of Lloyd's Algorithm with Distributed Local Iterations

TL;DR

This paper analyzes the convergence of Lloyd's K-means algorithm in a distributed setting with local iterations, showing the trade-off between communication cost and signal-to-noise ratio requirements.

Contribution

It introduces LocalKMeans, an algorithm for distributed K-means with local iterations, and provides theoretical analysis of its convergence and cost.

Findings

LocalKMeans performs Lloyd's algorithm with local steps on multiple machines.

Higher signal-to-noise ratio is required for convergence with local iterations.

The analysis adapts a virtual iterate method for non-convex, non-smooth objectives.

Abstract

In this paper, we analyze the classical $K$-means alternating-minimization algorithm, also known as Lloyd's algorithm (Lloyd, 1956), for a mixture of Gaussians in a data-distributed setting that incorporates local iteration steps. Assuming unlabeled data distributed across multiple machines, we propose an algorithm, LocalKMeans, that performs Lloyd's algorithm in parallel in the machines by running its iterations on local data, synchronizing only every $L$ of such local steps. We characterize the cost of these local iterations against the non-distributed setting, and show that the price paid for the local steps is a higher required signal-to-noise ratio. While local iterations were theoretically studied in the past for gradient-based learning methods, the analysis of unsupervised learning methods is more involved owing to the presence of latent variables, e.g. cluster identities, than that of an iterative gradient-based algorithm. To obtain our results, we adapt a virtual iterate method to work with a non-convex, non-smooth objective function, in conjunction with a tight statistical analysis of Lloyd steps.