Expert Level control of Ramp Metering based on Multi-task Deep Reinforcement Learning

TL;DR

This paper demonstrates how advanced deep reinforcement learning techniques can be applied to control complex cyberphysical systems, including multi-agent PDE systems, introducing novel algorithms for improved scalability and adaptability.

Contribution

It introduces the first use of RL for controlling systems modeled by discretized non-linear PDEs and proposes a new Mutual Weight Regularization algorithm for multi-agent control.

Findings

RL enables control of unknown, random, time-varying PDE parameters

The MWR algorithm improves scalability in multi-agent systems

Neural network RL effectively manages complex cyberphysical control tasks

Abstract

This article shows how the recent breakthroughs in Reinforcement Learning (RL) that have enabled robots to learn to play arcade video games, walk or assemble colored bricks, can be used to perform other tasks that are currently at the core of engineering cyberphysical systems. We present the first use of RL for the control of systems modeled by discretized non-linear Partial Differential Equations (PDEs) and devise a novel algorithm to use non-parametric control techniques for large multi-agent systems. We show how neural network based RL enables the control of discretized PDEs whose parameters are unknown, random, and time-varying. We introduce an algorithm of Mutual Weight Regularization (MWR) which alleviates the curse of dimensionality of multi-agent control schemes by sharing experience between agents while giving each agent the opportunity to specialize its action policy so as to tailor it to the local parameters of the part of the system it is located in.