Monkey Business: Reinforcement learning meets neighborhood search for Virtual Network Embedding

TL;DR

This paper introduces NEPA, a novel reinforcement learning algorithm combining neighborhood search with Monte Carlo Tree Search, to improve virtual network embedding for 5G slicing, achieving higher acceptance and revenue ratios.

Contribution

The paper proposes NEPA, a new algorithm that enhances NRPA with neighborhood search, effectively solving online virtual network embedding problems with improved performance.

Findings

NEPA outperforms state-of-the-art algorithms in acceptance ratio.

NEPA achieves higher revenue-to-cost ratios.

The approach is effective on both real and synthetic topologies.

Abstract

In this article, we consider the Virtual Network Embedding (VNE) problem for 5G networks slicing. This problem requires to allocate multiple Virtual Networks (VN) on a substrate virtualized physical network while maximizing among others, resource utilization, maximum number of placed VNs and network operator's benefit. We solve the online version of the problem where slices arrive over time. Inspired by the Nested Rollout Policy Adaptation (NRPA) algorithm, a variant of the well known Monte Carlo Tree Search (MCTS) that learns how to perform good simulations over time, we propose a new algorithm that we call Neighborhood Enhanced Policy Adaptation (NEPA). The key feature of our algorithm is to observe NRPA cannot exploit knowledge acquired in one branch of the state tree for another one which starts differently. NEPA learns by combining NRPA with Neighbordhood Search in a frugal manner which improves only promising solutions while keeping the running time low. We call this technique a monkey business because it comes down to jumping from one interesting branch to the other, similar to how monkeys jump from tree to tree instead of going down everytime. NEPA achieves better results in terms of acceptance ratio and revenue-to-cost ratio compared to other state-of-the-art algorithms, both on real and synthetic topologies.