Scalable Deep Reinforcement Learning for Routing and Spectrum Access in Physical Layer

TL;DR

This paper introduces a scalable deep reinforcement learning method for joint routing and spectrum access in wireless ad-hoc networks, accounting for physical-layer SINR and optimizing bottleneck metrics with a single agent per flow.

Contribution

It presents a novel scalable RL framework that jointly optimizes routing and spectrum access considering physical-layer SINR, unlike prior works with fixed topologies and separate tasks.

Findings

Effective optimization of bottleneck SINR along routes.

Single-agent approach improves scalability and generalizability.

Joint routing and spectrum access decision-making enhances network performance.

Abstract

This paper proposes a novel scalable reinforcement learning approach for simultaneous routing and spectrum access in wireless ad-hoc networks. In most previous works on reinforcement learning for network optimization, the network topology is assumed to be fixed, and a different agent is trained for each transmission node -- this limits scalability and generalizability. Further, routing and spectrum access are typically treated as separate tasks. Moreover, the optimization objective is usually a cumulative metric along the route, e.g., number of hops or delay. In this paper, we account for the physical-layer signal-to-interference-plus-noise ratio (SINR) in a wireless network and further show that bottleneck objective such as the minimum SINR along the route can also be optimized effectively using reinforcement learning. Specifically, we propose a scalable approach in which a single agent is associated with each flow and makes routing and spectrum access decisions as it moves along the frontier nodes. The agent is trained according to the physical-layer characteristics of the environment using a novel rewarding scheme based on the Monte Carlo estimation of the future bottleneck SINR. It learns to avoid interference by intelligently making joint routing and spectrum allocation decisions based on the geographical location information of the neighbouring nodes.