Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay

TL;DR

This paper introduces a novel reinforcement learning framework that combines DeepONet and backstepping control to effectively manage unstable PDEs with spatially-varying delays, outperforming traditional methods.

Contribution

It proposes a new DeepONet-based RL approach that eliminates the need for explicit delay function assumptions in controlling PDEs.

Findings

Outperforms baseline SAC without backstepping knowledge

Successfully controls unstable first-order hyperbolic PDEs with delays

Demonstrates effectiveness of combining analytical control with deep learning

Abstract

Control of distributed parameter systems affected by delays is a challenging task, particularly when the delays depend on spatial variables. The idea of integrating analytical control theory with learning-based control within a unified control scheme is becoming increasingly promising and advantageous. In this paper, we address the problem of controlling an unstable first-order hyperbolic PDE with spatially-varying delays by combining PDE backstepping control strategies and deep reinforcement learning (RL). To eliminate the assumption on the delay function required for the backstepping design, we propose a soft actor-critic (SAC) architecture incorporating a DeepONet to approximate the backstepping controller. The DeepONet extracts features from the backstepping controller and feeds them into the policy network. In simulations, our algorithm outperforms the baseline SAC without prior backstepping knowledge and the analytical controller.