Variational multiscale reinforcement learning for discovering reduced order closure models of nonlinear spatiotemporal transport systems

TL;DR

This paper introduces a variational multiscale reinforcement learning framework to develop robust, data-efficient closure models for nonlinear spatiotemporal systems, improving reduced order modeling accuracy without relying on high fidelity data.

Contribution

The study presents a novel variational multiscale RL approach that discovers closure models without high fidelity data, enhancing stability and accuracy in reduced order models of complex systems.

Findings

VMRL achieves robust closure modeling for viscous Burgers equation.

The method improves stability and accuracy over traditional approaches.

Potential to extend to other complex nonlinear dynamical systems.

Abstract

A central challenge in the computational modeling and simulation of a multitude of science applications is to achieve robust and accurate closures for their coarse-grained representations due to underlying highly nonlinear multiscale interactions. These closure models are common in many nonlinear spatiotemporal systems to account for losses due to reduced order representations, including many transport phenomena in fluids. Previous data-driven closure modeling efforts have mostly focused on supervised learning approaches using high fidelity simulation data. On the other hand, reinforcement learning (RL) is a powerful yet relatively uncharted method in spatiotemporally extended systems. In this study, we put forth a modular dynamic closure modeling and discovery framework to stabilize the Galerkin projection based reduced order models that may arise in many nonlinear spatiotemporal dynamical systems with quadratic nonlinearity. However, a key element in creating a robust RL agent is to introduce a feasible reward function, which can be constituted of any difference metrics between the RL model and high fidelity simulation data. First, we introduce a multi-modal RL (MMRL) to discover mode-dependant closure policies that utilize the high fidelity data in rewarding our RL agent. We then formulate a variational multiscale RL (VMRL) approach to discover closure models without requiring access to the high fidelity data in designing the reward function. Specifically, our chief innovation is to leverage variational multiscale formalism to quantify the difference between modal interactions in Galerkin systems. Our results in simulating the viscous Burgers equation indicate that the proposed VMRL method leads to robust and accurate closure parameterizations, and it may potentially be used to discover scale-aware closure models for complex dynamical systems.