Advanced Scaling Methods for VNF deployment with Reinforcement Learning

TL;DR

This paper introduces an advanced reinforcement learning approach using improved graph neural networks and phasic policy gradient to optimize VNF deployment, achieving better QoS and resource efficiency across diverse network scenarios.

Contribution

It presents a novel RL model with enhanced GNN architecture and PPG, improving generalization and deployment efficiency for complex network environments.

Findings

Achieves better QoS with minimal resource use

Demonstrates improved generalization across scenarios

Provides more interpretable node representations

Abstract

Network function virtualization (NFV) and software-defined network (SDN) have become emerging network paradigms, allowing virtualized network function (VNF) deployment at a low cost. Even though VNF deployment can be flexible, it is still challenging to optimize VNF deployment due to its high complexity. Several studies have approached the task as dynamic programming, e.g., integer linear programming (ILP). However, optimizing VNF deployment for highly complex networks remains a challenge. Alternatively, reinforcement learning (RL) based approaches have been proposed to optimize this task, especially to employ a scaling action-based method which can deploy VNFs within less computational time. However, the model architecture can be improved further to generalize to the different networking settings. In this paper, we propose an enhanced model which can be adapted to more general network settings. We adopt the improved GNN architecture and a few techniques to obtain a better node representation for the VNF deployment task. Furthermore, we apply a recently proposed RL method, phasic policy gradient (PPG), to leverage the shared representation of the service function chain (SFC) generation model from the value function. We evaluate the proposed method in various scenarios, achieving a better QoS with minimum resource utilization compared to the previous methods. Finally, as a qualitative evaluation, we analyze our proposed encoder's representation for the nodes, which shows a more disentangled representation.