Age of Information Aware VNF Scheduling in Industrial IoT Using Deep Reinforcement Learning

TL;DR

This paper proposes a deep reinforcement learning approach for VNF scheduling in industrial IoT, optimizing information freshness and network costs, with multi-agent collaboration improving efficiency over single-agent methods.

Contribution

It introduces a novel DRL-based VNF scheduling framework that jointly minimizes AoI and network costs, extending from single-agent to multi-agent systems for better performance.

Findings

Single-agent DRL outperforms greedy algorithms in cost and AoI.

Multi-agent DRL reduces average network cost through task division.

Multi-agent approach requires more training iterations due to collaboration complexity.

Abstract

In delay-sensitive industrial internet of things (IIoT) applications, the age of information (AoI) is employed to characterize the freshness of information. Meanwhile, the emerging network function virtualization provides flexibility and agility for service providers to deliver a given network service using a sequence of virtual network functions (VNFs). However, suitable VNF placement and scheduling in these schemes is NP-hard and finding a globally optimal solution by traditional approaches is complex. Recently, deep reinforcement learning (DRL) has appeared as a viable way to solve such problems. In this paper, we first utilize single agent low-complex compound action actor-critic RL to cover both discrete and continuous actions and jointly minimize VNF cost and AoI in terms of network resources under end-to end Quality of Service constraints. To surmount the single-agent capacity limitation for learning, we then extend our solution to a multi-agent DRL scheme in which agents collaborate with each other. Simulation results demonstrate that single-agent schemes significantly outperform the greedy algorithm in terms of average network cost and AoI. Moreover, multi-agent solution decreases the average cost by dividing the tasks between the agents. However, it needs more iterations to be learned due to the requirement on the agents collaboration.