Flexible Parallel Learning in Edge Scenarios: Communication, Computational and Energy Cost

TL;DR

This paper introduces a flexible parallel learning framework for edge scenarios that combines data and model parallelism, optimizing computation, communication, and energy costs for IoT and fog environments.

Contribution

The work presents a novel framework for flexible parallel learning that integrates data and model parallelism tailored for edge computing scenarios.

Findings

FPL achieves a good trade-off between energy, communication, and performance.

Experiments with deep networks and large datasets validate FPL's efficiency.

FPL adapts to diverse distributed learning requirements in IoT environments.

Abstract

Traditionally, distributed machine learning takes the guise of (i) different nodes training the same model (as in federated learning), or (ii) one model being split among multiple nodes (as in distributed stochastic gradient descent). In this work, we highlight how fog- and IoT-based scenarios often require combining both approaches, and we present a framework for flexible parallel learning (FPL), achieving both data and model parallelism. Further, we investigate how different ways of distributing and parallelizing learning tasks across the participating nodes result in different computation, communication, and energy costs. Our experiments, carried out using state-of-the-art deep-network architectures and large-scale datasets, confirm that FPL allows for an excellent trade-off among computational (hence energy) cost, communication overhead, and learning performance.