Analysis of Independent Learning in Network Agents: A Packet Forwarding Use Case

TL;DR

This paper evaluates independent multi-agent reinforcement learning strategies for network packet forwarding, highlighting their benefits, challenges, and performance issues within the Named Data Networking architecture.

Contribution

It provides a comprehensive assessment of IQL-based independent learning approaches for packet forwarding, including performance analysis and identification of key challenges.

Findings

Independent IQL strategies can be scalable but face performance challenges.

Model tuning and network topology significantly impact IDQF effectiveness.

IDQF performance issues are linked to training and testing isolation problems.

Abstract

Multi-Agent Reinforcement Learning (MARL) is nowadays widely used to solve real-world and complex decisions in various domains. While MARL can be categorized into independent and cooperative approaches, we consider the independent approach as a simple, more scalable, and less costly method for large-scale distributed systems, such as network packet forwarding. In this paper, we quantitatively and qualitatively assess the benefits of leveraging such independent agents learning approach, in particular IQL-based algorithm, for packet forwarding in computer networking, using the Named Data Networking (NDN) architecture as a driving example. We put multiple IQL-based forwarding strategies (IDQF) to the test and compare their performances against very basic forwarding schemes and simple topologies/traffic models to highlight major challenges and issues. We discuss the main issues related to the poor performance of IDQF and quantify the impact of these issues on isolation when training and testing the IDQF models under different model tuning parameters and network topologies/characteristics.