Model-Based Reinforcement Learning for Control of Strongly-Disturbed Unsteady Aerodynamic Flows

TL;DR

This paper introduces a model-based reinforcement learning approach using a physics-augmented autoencoder and latent dynamics model to control unsteady aerodynamic flows efficiently, even under strong disturbances.

Contribution

It develops a novel reduced-order model integrating physics and deep learning for effective control of complex aerodynamic flows.

Findings

The model accurately predicts flow dynamics in disturbed environments.

The learned control policy effectively mitigates lift variation during gusts.

The approach reduces training costs compared to model-free RL methods.

Abstract

The intrinsic high dimension of fluid dynamics is an inherent challenge to control of aerodynamic flows, and this is further complicated by a flow's nonlinear response to strong disturbances. Deep reinforcement learning, which takes advantage of the exploratory aspects of reinforcement learning (RL) and the rich nonlinearity of a deep neural network, provides a promising approach to discover feasible control strategies. However, the typical model-free approach to reinforcement learning requires a significant amount of interaction between the flow environment and the RL agent during training, and this high training cost impedes its development and application. In this work, we propose a model-based reinforcement learning (MBRL) approach by incorporating a novel reduced-order model as a surrogate for the full environment. The model consists of a physics-augmented autoencoder, which compresses high-dimensional CFD flow field snaphsots into a three-dimensional latent space, and a latent dynamics model that is trained to accurately predict the long-time dynamics of trajectories in the latent space in response to action sequences. The accuracy and robustness of the model are demonstrated in the scenario of a pitching airfoil within a highly disturbed environment. Additionally, an application to a vertical-axis wind turbine in a disturbance-free environment is discussed in the Appendix Based on the model trained in the pitching airfoil problem, we realize an MBRL strategy to mitigate lift variation during gust-airfoil encounters. We demonstrate that the policy learned in the reduced-order environment translates to an effective control strategy in the full CFD environment.