Homogeneous Learning: Self-Attention Decentralized Deep Learning

TL;DR

Homogeneous Learning introduces a decentralized deep learning approach using self-attention and reinforcement learning to efficiently handle non-IID data, reducing training rounds and communication costs without a central server.

Contribution

This paper presents a novel decentralized learning model called Homogeneous Learning that employs self-attention and reinforcement learning for node selection, improving efficiency on non-IID data.

Findings

Reduces total training rounds by 50.8%.

Decreases communication cost by 74.6%.

Outperforms standalone learning in image classification tasks.

Abstract

Federated learning (FL) has been facilitating privacy-preserving deep learning in many walks of life such as medical image classification, network intrusion detection, and so forth. Whereas it necessitates a central parameter server for model aggregation, which brings about delayed model communication and vulnerability to adversarial attacks. A fully decentralized architecture like Swarm Learning allows peer-to-peer communication among distributed nodes, without the central server. One of the most challenging issues in decentralized deep learning is that data owned by each node are usually non-independent and identically distributed (non-IID), causing time-consuming convergence of model training. To this end, we propose a decentralized learning model called Homogeneous Learning (HL) for tackling non-IID data with a self-attention mechanism. In HL, training performs on each round's selected node, and the trained model of a node is sent to the next selected node at the end of each round. Notably, for the selection, the self-attention mechanism leverages reinforcement learning to observe a node's inner state and its surrounding environment's state, and find out which node should be selected to optimize the training. We evaluate our method with various scenarios for an image classification task. The result suggests that HL can produce a better performance compared with standalone learning and greatly reduce both the total training rounds by 50.8% and the communication cost by 74.6% compared with random policy-based decentralized learning for training on non-IID data.