# Optimal Network Control in Partially-Controllable Networks

## TL;DR

This paper develops new algorithms for optimal network control in partially-controllable networks, addressing unknown policies of uncontrollable nodes and balancing throughput, delay, and convergence.

## Contribution

It introduces TMW, a low-complexity algorithm for queue-agnostic policies, and TUCRL, a reinforcement learning method for queue-dependent policies, both tailored for partially-controllable networks.

## Key findings

- TMW is throughput-optimal in queue-agnostic scenarios.
- TUCRL achieves tunable tradeoffs among throughput, delay, and convergence.
- The algorithms perform effectively in models with unknown uncontrollable node policies.

## Abstract

The effectiveness of many optimal network control algorithms (e.g., BackPressure) relies on the premise that all of the nodes are fully controllable. However, these algorithms may yield poor performance in a partially-controllable network where a subset of nodes are uncontrollable and use some unknown policy. Such a partially-controllable model is of increasing importance in real-world networked systems such as overlay-underlay networks. In this paper, we design optimal network control algorithms that can stabilize a partially-controllable network. We first study the scenario where uncontrollable nodes use a queue-agnostic policy, and propose a low-complexity throughput-optimal algorithm, called Tracking-MaxWeight (TMW), which enhances the original MaxWeight algorithm with an explicit learning of the policy used by uncontrollable nodes. Next, we investigate the scenario where uncontrollable nodes use a queue-dependent policy and the problem is formulated as an MDP with unknown queueing dynamics. We propose a new reinforcement learning algorithm, called Truncated Upper Confidence Reinforcement Learning (TUCRL), and prove that TUCRL achieves tunable three-way tradeoffs between throughput, delay and convergence rate.