Decoupled Contrastive Learning for Federated Learning

TL;DR

This paper introduces Decoupled Contrastive Learning for Federated Learning (DCFL), a novel approach that improves model training by decoupling contrastive loss components, effectively handling data heterogeneity and small sample sizes in federated settings.

Contribution

The paper proposes a new framework, DCFL, that decouples contrastive loss into alignment and uniformity components, enabling better calibration for federated learning with limited data per client.

Findings

DCFL achieves stronger alignment and uniformity than existing methods.

Experimental results on CIFAR-10, CIFAR-100, and Tiny-ImageNet show DCFL outperforms state-of-the-art federated learning methods.

Abstract

Federated learning is a distributed machine learning paradigm that allows multiple participants to train a shared model by exchanging model updates instead of their raw data. However, its performance is degraded compared to centralized approaches due to data heterogeneity across clients. While contrastive learning has emerged as a promising approach to mitigate this, our theoretical analysis reveals a fundamental conflict: its asymptotic assumptions of an infinite number of negative samples are violated in finite-sample regime of federated learning. To address this issue, we introduce Decoupled Contrastive Learning for Federated Learning (DCFL), a novel framework that decouples the existing contrastive loss into two objectives. Decoupling the loss into its alignment and uniformity components enables the independent calibration of the attraction and repulsion forces without relying on the asymptotic assumptions. This strategy provides a contrastive learning method suitable for federated learning environments where each client has a small amount of data. Our experimental results show that DCFL achieves stronger alignment between positive samples and greater uniformity between negative samples compared to existing contrastive learning methods. Furthermore, experimental results on standard benchmarks, including CIFAR-10, CIFAR-100, and Tiny-ImageNet, demonstrate that DCFL consistently outperforms state-of-the-art federated learning methods.