Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics

TL;DR

This paper demonstrates that applying symmetry reduction to deep reinforcement learning significantly improves control efficiency and robustness in managing chaotic spatiotemporal dynamics, exemplified by the Kuramoto-Sivashinsky Equation.

Contribution

The authors introduce a symmetry-aware deep RL framework that enhances data efficiency and control performance in flow control tasks involving symmetric high-dimensional systems.

Findings

Symmetry reduction improves data efficiency in deep RL for flow control.

Symmetry-aware policies stabilize equilibrium states of the system.

Control policies are robust to noise and unseen system parameters.

Abstract

Deep reinforcement learning (RL) is a data-driven, model-free method capable of discovering complex control strategies for macroscopic objectives in high-dimensional systems, making its application towards flow control promising. Many systems of flow control interest possess symmetries that, when neglected, can significantly inhibit the learning and performance of a naive deep RL approach. Using a test-bed consisting of the Kuramoto-Sivashinsky Equation (KSE), equally spaced actuators, and a goal of minimizing dissipation and power cost, we demonstrate that by moving the deep RL problem to a symmetry-reduced space, we can alleviate limitations inherent in the naive application of deep RL. We demonstrate that symmetry-reduced deep RL yields improved data efficiency as well as improved control policy efficacy compared to policies found by naive deep RL. Interestingly, the policy learned by the the symmetry aware control agent drives the system toward an equilibrium state of the forced KSE that is connected by continuation to an equilibrium of the unforced KSE, despite having been given no explicit information regarding its existence. I.e., to achieve its goal, the RL algorithm discovers and stabilizes an equilibrium state of the system. Finally, we demonstrate that the symmetry-reduced control policy is robust to observation and actuation signal noise, as well as to system parameters it has not observed before.