Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?

TL;DR

This paper investigates the effectiveness of self-supervised learning in decentralized, unlabeled, and heterogeneous data settings, demonstrating robustness and efficiency in learning useful visual representations.

Contribution

It provides a comprehensive study of decentralized SSL algorithms on large-scale datasets, introduces a new Dec-SSL algorithm with improved performance, and offers theoretical insights into data heterogeneity effects.

Findings

Decentralized SSL is robust to data heterogeneity.

Dec-SSL reduces communication and participation costs.

Dec-SSL can outperform decentralized supervised learning.

Abstract

Decentralized learning has been advocated and widely deployed to make efficient use of distributed datasets, with an extensive focus on supervised learning (SL) problems. Unfortunately, the majority of real-world data are unlabeled and can be highly heterogeneous across sources. In this work, we carefully study decentralized learning with unlabeled data through the lens of self-supervised learning (SSL), specifically contrastive visual representation learning. We study the effectiveness of a range of contrastive learning algorithms under decentralized learning settings, on relatively large-scale datasets including ImageNet-100, MS-COCO, and a new real-world robotic warehouse dataset. Our experiments show that the decentralized SSL (Dec-SSL) approach is robust to the heterogeneity of decentralized datasets, and learns useful representation for object classification, detection, and segmentation tasks. This robustness makes it possible to significantly reduce communication and reduce the participation ratio of data sources with only minimal drops in performance. Interestingly, using the same amount of data, the representation learned by Dec-SSL can not only perform on par with that learned by centralized SSL which requires communication and excessive data storage costs, but also sometimes outperform representations extracted from decentralized SL which requires extra knowledge about the data labels. Finally, we provide theoretical insights into understanding why data heterogeneity is less of a concern for Dec-SSL objectives, and introduce feature alignment and clustering techniques to develop a new Dec-SSL algorithm that further improves the performance, in the face of highly non-IID data. Our study presents positive evidence to embrace unlabeled data in decentralized learning, and we hope to provide new insights into whether and why decentralized SSL is effective.