Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering

TL;DR

Orchestra introduces a novel unsupervised federated learning method that uses hierarchical clustering to achieve globally consistent data partitions, improving robustness, scalability, and communication efficiency across diverse client conditions.

Contribution

It proposes a new federated learning approach leveraging hierarchical clustering for global consistency, addressing limitations of existing self-supervised methods.

Findings

Outperforms alternative techniques across various heterogeneity levels

Maintains good generalization under linear probing

Effective with different numbers of clients and participation ratios

Abstract

Federated learning is generally used in tasks where labels are readily available (e.g., next word prediction). Relaxing this constraint requires design of unsupervised learning techniques that can support desirable properties for federated training: robustness to statistical/systems heterogeneity, scalability with number of participants, and communication efficiency. Prior work on this topic has focused on directly extending centralized self-supervised learning techniques, which are not designed to have the properties listed above. To address this situation, we propose Orchestra, a novel unsupervised federated learning technique that exploits the federation's hierarchy to orchestrate a distributed clustering task and enforce a globally consistent partitioning of clients' data into discriminable clusters. We show the algorithmic pipeline in Orchestra guarantees good generalization performance under a linear probe, allowing it to outperform alternative techniques in a broad range of conditions, including variation in heterogeneity, number of clients, participation ratio, and local epochs.