On the effects of similarity metrics in decentralized deep learning under distributional shift

TL;DR

This paper investigates how different similarity metrics impact decentralized deep learning performance under data heterogeneity and distributional shifts, providing empirical insights for better peer selection.

Contribution

It offers an empirical analysis of various similarity metrics in decentralized learning, highlighting their effectiveness and limitations in heterogeneous data scenarios.

Findings

Certain similarity metrics improve model merging under distributional shift

Some metrics outperform others in heterogeneous data environments

Insights guide robust decentralized learning methods

Abstract

Decentralized Learning (DL) enables privacy-preserving collaboration among organizations or users to enhance the performance of local deep learning models. However, model aggregation becomes challenging when client data is heterogeneous, and identifying compatible collaborators without direct data exchange remains a pressing issue. In this paper, we investigate the effectiveness of various similarity metrics in DL for identifying peers for model merging, conducting an empirical analysis across multiple datasets with distribution shifts. Our research provides insights into the performance of these metrics, examining their role in facilitating effective collaboration. By exploring the strengths and limitations of these metrics, we contribute to the development of robust DL methods.