Soft Separation and Distillation: Toward Global Uniformity in Federated Unsupervised Learning

TL;DR

This paper introduces Soft Separation and Distillation (SSD), a novel method to improve global uniformity of representations in federated unsupervised learning, addressing non-IID data challenges and enhancing overall model performance.

Contribution

SSD is a new approach that promotes inter-client uniformity in federated unsupervised learning, reducing interference during aggregation and improving representation quality.

Findings

SSD improves global uniformity in federated learning.

SSD enhances representation quality and task performance.

SSD is effective in both cross-silo and cross-device settings.

Abstract

Federated Unsupervised Learning (FUL) aims to learn expressive representations in federated and self-supervised settings. The quality of representations learned in FUL is usually determined by uniformity, a measure of how uniformly representations are distributed in the embedding space. However, existing solutions perform well in achieving intra-client (local) uniformity for local models while failing to achieve inter-client (global) uniformity after aggregation due to non-IID data distributions and the decentralized nature of FUL. To address this issue, we propose Soft Separation and Distillation (SSD), a novel approach that preserves inter-client uniformity by encouraging client representations to spread toward different directions. This design reduces interference during client model aggregation, thereby improving global uniformity while preserving local representation expressiveness. We further enhance this effect by introducing a projector distillation module to address the discrepancy between loss optimization and representation quality. We evaluate SSD in both cross-silo and cross-device federated settings, demonstrating consistent improvements in representation quality and task performance across various training scenarios. Our results highlight the importance of inter-client uniformity in FUL and establish SSD as an effective solution to this challenge. Project page: https://ssd-uniformity.github.io/