Interpretable and Efficient Data-driven Discovery and Control of Distributed Systems

TL;DR

This paper introduces a scalable, interpretable, and data-efficient model-based reinforcement learning framework for controlling complex PDE-governed systems, validated on fluid flow problems.

Contribution

It combines SINDy-C and autoencoders to create an interpretable, scalable RL approach for PDE control, addressing sample inefficiency and interpretability issues.

Findings

Successfully applied to 1D Burgers and 2D Navier-Stokes equations.

Achieved faster learning with fewer environment interactions.

Provided interpretable latent space dynamics.

Abstract

Effectively controlling systems governed by Partial Differential Equations (PDEs) is crucial in several fields of Applied Sciences and Engineering. These systems usually yield significant challenges to conventional control schemes due to their nonlinear dynamics, partial observability, high-dimensionality once discretized, distributed nature, and the requirement for low-latency feedback control. Reinforcement Learning (RL), particularly Deep RL (DRL), has recently emerged as a promising control paradigm for such systems, demonstrating exceptional capabilities in managing high-dimensional, nonlinear dynamics. However, DRL faces challenges including sample inefficiency, robustness issues, and an overall lack of interpretability. To address these issues, we propose a data-efficient, interpretable, and scalable Dyna-style Model-Based RL framework for PDE control, combining the Sparse Identification of Nonlinear Dynamics with Control (SINDy-C) algorithm and an autoencoder (AE) framework for the sake of dimensionality reduction of PDE states and actions. This novel approach enables fast rollouts, reducing the need for extensive environment interactions, and provides an interpretable latent space representation of the PDE forward dynamics. We validate our method on two PDE problems describing fluid flows - namely, the 1D Burgers equation and 2D Navier-Stokes equations - comparing it against a model-free baseline, and carrying out an extensive analysis of the learned dynamics.