DISTREAL: Distributed Resource-Aware Learning in Heterogeneous Systems

TL;DR

DISTREAL introduces a resource-aware distributed training method for neural networks that dynamically adjusts computational complexity via dropout, optimizing convergence speed in heterogeneous, resource-constrained environments.

Contribution

It presents a novel design space exploration technique to find Pareto-optimal dropout configurations per layer, enabling devices to adaptively utilize available resources without server assistance.

Findings

Significantly faster convergence compared to existing methods.

Maintains high final accuracy despite resource variability.

Effective in federated learning with heterogeneous device resources.

Abstract

We study the problem of distributed training of neural networks (NNs) on devices with heterogeneous, limited, and time-varying availability of computational resources. We present an adaptive, resource-aware, on-device learning mechanism, DISTREAL, which is able to fully and efficiently utilize the available resources on devices in a distributed manner, increasing the convergence speed. This is achieved with a dropout mechanism that dynamically adjusts the computational complexity of training an NN by randomly dropping filters of convolutional layers of the model. Our main contribution is the introduction of a design space exploration (DSE) technique, which finds Pareto-optimal per-layer dropout vectors with respect to resource requirements and convergence speed of the training. Applying this technique, each device is able to dynamically select the dropout vector that fits its available resource without requiring any assistance from the server. We implement our solution in a federated learning (FL) system, where the availability of computational resources varies both between devices and over time, and show through extensive evaluation that we are able to significantly increase the convergence speed over the state of the art without compromising on the final accuracy.