FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Data

TL;DR

FedSC introduces a provable federated self-supervised learning algorithm that leverages spectral contrastive objectives and shared correlation matrices to improve data representation quality in non-i.i.d. settings, with privacy guarantees.

Contribution

The paper proposes FedSC, a novel federated SSL algorithm using spectral contrastive objectives and correlation matrix sharing, with theoretical convergence and privacy analysis.

Findings

FedSC outperforms baseline methods in non-i.i.d. data scenarios.

Sharing correlation matrices enhances data representation quality.

Theoretical analysis confirms convergence and privacy guarantees.

Abstract

Recent efforts have been made to integrate self-supervised learning (SSL) with the framework of federated learning (FL). One unique challenge of federated self-supervised learning (FedSSL) is that the global objective of FedSSL usually does not equal the weighted sum of local SSL objectives. Consequently, conventional approaches, such as federated averaging (FedAvg), fail to precisely minimize the FedSSL global objective, often resulting in suboptimal performance, especially when data is non-i.i.d.. To fill this gap, we propose a provable FedSSL algorithm, named FedSC, based on the spectral contrastive objective. In FedSC, clients share correlation matrices of data representations in addition to model weights periodically, which enables inter-client contrast of data samples in addition to intra-client contrast and contraction, resulting in improved quality of data representations. Differential privacy (DP) protection is deployed to control the additional privacy leakage on local datasets when correlation matrices are shared. We also provide theoretical analysis on the convergence and extra privacy leakage. The experimental results validate the effectiveness of our proposed algorithm.