Holarchic Structures for Decentralized Deep Learning - A Performance Analysis

TL;DR

This paper introduces holarchic structures for decentralized deep learning, demonstrating through extensive experiments that they improve performance and robustness in IoT environments with network uncertainties.

Contribution

It proposes a novel self-adaptive holarchic learning framework tailored for decentralized deep learning in IoT networks, addressing network uncertainties and resource constraints.

Findings

Holarchic structures significantly improve learning performance in decentralized systems.

The approach is cost-effective and scalable based on large-scale experiments.

Performance gains are confirmed with real-world data from smart city projects.

Abstract

Structure plays a key role in learning performance. In centralized computational systems, hyperparameter optimization and regularization techniques such as dropout are computational means to enhance learning performance by adjusting the deep hierarchical structure. However, in decentralized deep learning by the Internet of Things, the structure is an actual network of autonomous interconnected devices such as smart phones that interact via complex network protocols. Self-adaptation of the learning structure is a challenge. Uncertainties such as network latency, node and link failures or even bottlenecks by limited processing capacity and energy availability can signif- icantly downgrade learning performance. Network self-organization and self-management is complex, while it requires additional computational and network resources that hinder the feasibility of decentralized deep learning. In contrast, this paper introduces a self-adaptive learning approach based on holarchic learning structures for exploring, mitigating and boosting learning performance in distributed environments with uncertainties. A large-scale performance analysis with 864000 experiments fed with synthetic and real-world data from smart grid and smart city pilot projects confirm the cost-effectiveness of holarchic structures for decentralized deep learning.