Enabling Deep Reinforcement Learning on Energy Constrained Devices at the Edge of the Network

TL;DR

This paper presents a method for deploying deep reinforcement learning on energy-harvesting edge devices by training the model centrally and periodically updating the device, maintaining performance with minimal energy use.

Contribution

It introduces a two-part algorithm that trains DRL centrally and updates embedded devices periodically, enabling energy-efficient operation on energy-harvesting devices.

Findings

DRL can operate effectively on energy-harvesting devices with minimal updates.

Periodic weight transfer maintains performance without degradation.

The approach is validated with real-world data and AoI optimization.

Abstract

Deep Reinforcement Learning (DRL) solutions are becoming pervasive at the edge of the network as they enable autonomous decision-making in a dynamic environment. However, to be able to adapt to the ever-changing environment, the DRL solution implemented on an embedded device has to continue to occasionally take exploratory actions even after initial convergence. In other words, the device has to occasionally take random actions and update the value function, i.e., re-train the Artificial Neural Network (ANN), to ensure its performance remains optimal. Unfortunately, embedded devices often lack processing power and energy required to train the ANN. The energy aspect is particularly challenging when the edge device is powered only by a means of Energy Harvesting (EH). To overcome this problem, we propose a two-part algorithm in which the DRL process is trained at the sink. Then the weights of the fully trained underlying ANN are periodically transferred to the EH-powered embedded device taking actions. Using an EH-powered sensor, real-world measurements dataset, and optimizing for Age of Information (AoI) metric, we demonstrate that such a DRL solution can operate without any degradation in the performance, with only a few ANN updates per day.