A Distributed Deep Reinforcement Learning Technique for Application Placement in Edge and Fog Computing Environments

TL;DR

This paper introduces a distributed deep reinforcement learning method for application placement in edge and fog computing, addressing the limitations of centralized approaches by improving adaptability and efficiency for complex IoT application topologies.

Contribution

It proposes an actor-critic-based distributed placement technique using IMPALA, enhancing scalability, sample efficiency, and convergence speed for IoT application placement.

Findings

Achieves up to 30% reduction in execution cost.

Demonstrates improved scalability over centralized methods.

Effectively handles DAG-based IoT applications.

Abstract

Fog/Edge computing is a novel computing paradigm supporting resource-constrained Internet of Things (IoT) devices by the placement of their tasks on the edge and/or cloud servers. Recently, several Deep Reinforcement Learning (DRL)-based placement techniques have been proposed in fog/edge computing environments, which are only suitable for centralized setups. The training of well-performed DRL agents requires manifold training data while obtaining training data is costly. Hence, these centralized DRL-based techniques lack generalizability and quick adaptability, thus failing to efficiently tackle application placement problems. Moreover, many IoT applications are modeled as Directed Acyclic Graphs (DAGs) with diverse topologies. Satisfying dependencies of DAG-based IoT applications incur additional constraints and increase the complexity of placement problems. To overcome these challenges, we propose an actor-critic-based distributed application placement technique, working based on the IMPortance weighted Actor-Learner Architectures (IMPALA). IMPALA is known for efficient distributed experience trajectory generation that significantly reduces the exploration costs of agents. Besides, it uses an adaptive off-policy correction method for faster convergence to optimal solutions. Our technique uses recurrent layers to capture temporal behaviors of input data and a replay buffer to improve the sample efficiency. The performance results, obtained from simulation and testbed experiments, demonstrate that our technique significantly improves the execution cost of IoT applications up to 30\% compared to its counterparts.