Topology-Aware Knowledge Propagation in Decentralized Learning

TL;DR

This paper investigates how knowledge, especially out-of-distribution data, propagates in decentralized learning systems and proposes topology-aware strategies to improve this propagation, significantly enhancing model accuracy.

Contribution

The paper identifies challenges in propagating out-of-distribution knowledge in decentralized learning and introduces topology-aware aggregation methods to address these issues.

Findings

Topology-aware strategies improve OOD knowledge propagation by 123% on average.

Propagation of OOD knowledge is heavily influenced by topology and data location.

Popular algorithms struggle with effective OOD knowledge dissemination across devices.

Abstract

Decentralized learning enables collaborative training of models across naturally distributed data without centralized coordination or maintenance of a global model. Instead, devices are organized in arbitrary communication topologies, in which they can only communicate with neighboring devices. Each device maintains its own local model by training on its local data and integrating new knowledge via model aggregation with neighbors. Therefore, knowledge is propagated across the topology via successive aggregation rounds. We study, in particular, the propagation of out-of-distribution (OOD) knowledge. We find that popular decentralized learning algorithms struggle to propagate OOD knowledge effectively to all devices. Further, we find that both the location of OOD data within a topology, and the topology itself, significantly impact OOD knowledge propagation. We then propose topology-aware aggregation strategies to accelerate (OOD) knowledge propagation across devices. These strategies improve OOD data accuracy, compared to topology-unaware baselines, by 123% on average across models in a topology.