Dynamic Preference Multi-Objective Reinforcement Learning for Internet Network Management

TL;DR

This paper introduces a reinforcement learning approach for internet network management that dynamically adapts to changing preferences, improving generalization across various network states and preference scenarios.

Contribution

The paper proposes a novel RL-based method allowing network management agents to adapt actions based on both states and varying preferences, with a numerical technique to estimate preference distributions for unbiased training.

Findings

Agents trained with the new approach outperform static preference models.

The numerical estimation method enhances training stability and bias reduction.

The approach improves generalization across diverse network conditions.

Abstract

An internet network service provider manages its network with multiple objectives, such as high quality of service (QoS) and minimum computing resource usage. To achieve these objectives, a reinforcement learning-based (RL) algorithm has been proposed to train its network management agent. Usually, their algorithms optimize their agents with respect to a single static reward formulation consisting of multiple objectives with fixed importance factors, which we call preferences. However, in practice, the preference could vary according to network status, external concerns and so on. For example, when a server shuts down and it can cause other servers' traffic overloads leading to additional shutdowns, it is plausible to reduce the preference of QoS while increasing the preference of minimum computing resource usages. In this paper, we propose new RL-based network management agents that can select actions based on both states and preferences. With our proposed approach, we expect a single agent to generalize on various states and preferences. Furthermore, we propose a numerical method that can estimate the distribution of preference that is advantageous for unbiased training. Our experiment results show that the RL agents trained based on our proposed approach significantly generalize better with various preferences than the previous RL approaches, which assume static preference during training. Moreover, we demonstrate several analyses that show the advantages of our numerical estimation method.