Federated Double Deep Q-learning for Joint Delay and Energy Minimization in IoT networks

TL;DR

This paper introduces a federated deep reinforcement learning framework using double deep Q-networks to optimize delay and energy consumption in IoT networks, enhancing learning speed and scalability.

Contribution

It presents a novel federated DDQN approach for multi-objective IoT optimization, addressing privacy and scalability issues in distributed reinforcement learning.

Findings

Federated DDQN outperforms federated DQN and non-federated DDQN in learning speed.

Incorporating federated learning improves scalability and privacy.

Optimization of network parameters affects learning efficiency.

Abstract

In this paper, we propose a federated deep reinforcement learning framework to solve a multi-objective optimization problem, where we consider minimizing the expected long-term task completion delay and energy consumption of IoT devices. This is done by optimizing offloading decisions, computation resource allocation, and transmit power allocation. Since the formulated problem is a mixed-integer non-linear programming (MINLP), we first cast our problem as a multi-agent distributed deep reinforcement learning (DRL) problem and address it using double deep Q-network (DDQN), where the actions are offloading decisions. The immediate cost of each agent is calculated through solving either the transmit power optimization or local computation resource optimization, based on the selected offloading decisions (actions). Then, to enhance the learning speed of IoT devices (agents), we incorporate federated learning (FDL) at the end of each episode. FDL enhances the scalability of the proposed DRL framework, creates a context for cooperation between agents, and minimizes their privacy concerns. Our numerical results demonstrate the efficacy of our proposed federated DDQN framework in terms of learning speed compared to federated deep Q network (DQN) and non-federated DDQN algorithms. In addition, we investigate the impact of batch size, network layers, DDQN target network update frequency on the learning speed of the FDL.