Edge Delayed Deep Deterministic Policy Gradient: efficient continuous control for edge scenarios

TL;DR

This paper introduces EdgeD3, an efficient reinforcement learning algorithm optimized for edge scenarios, reducing computational resources by 30% while maintaining or improving performance compared to existing methods.

Contribution

EdgeD3 is a novel extension of DDPG that significantly reduces GPU time and memory usage, tailored for resource-constrained edge environments.

Findings

Achieves 25% less GPU time than standard DDPG.

Uses 30% less memory and computational resources.

Outperforms or matches state-of-the-art methods on benchmarks.

Abstract

Deep Reinforcement Learning is gaining increasing attention thanks to its capability to learn complex policies in high-dimensional settings. Recent advancements utilize a dual-network architecture to learn optimal policies through the Q-learning algorithm. However, this approach has notable drawbacks, such as an overestimation bias that can disrupt the learning process and degrade the performance of the resulting policy. To address this, novel algorithms have been developed that mitigate overestimation bias by employing multiple Q-functions. Edge scenarios, which prioritize privacy, have recently gained prominence. In these settings, limited computational resources pose a significant challenge for complex Machine Learning approaches, making the efficiency of algorithms crucial for their performance. In this work, we introduce a novel Reinforcement Learning algorithm tailored for edge scenarios, called Edge Delayed Deep Deterministic Policy Gradient (EdgeD3). EdgeD3 enhances the Deep Deterministic Policy Gradient (DDPG) algorithm, achieving significantly improved performance with $25\%$ less Graphics Process Unit (GPU) time while maintaining the same memory usage. Additionally, EdgeD3 consistently matches or surpasses the performance of state-of-the-art methods across various benchmarks, all while using $30\%$ fewer computational resources and requiring $30\%$ less memory.