A Stochastic Maximum Principle Approach for Reinforcement Learning with Parameterized Environment

TL;DR

This paper presents a novel stochastic maximum principle approach for reinforcement learning that leverages physics-based parameterization of environments, enabling efficient policy learning with fewer training episodes.

Contribution

It introduces a new SMP-based framework for RL with environment parameterization, combining online parameter estimation and backward action learning for improved efficiency.

Findings

Requires fewer training episodes than dynamic programming methods.

Produces reliable control policies in numerical experiments.

Integrates physics knowledge into RL for better environment modeling.

Abstract

In this work, we introduce a stochastic maximum principle (SMP) approach for solving the reinforcement learning problem with the assumption that the unknowns in the environment can be parameterized based on physics knowledge. For the development of numerical algorithms, we shall apply an effective online parameter estimation method as our exploration technique to estimate the environment parameter during the training procedure, and the exploitation for the optimal policy will be achieved by an efficient backward action learning method for policy improvement under the SMP framework. Numerical experiments will be presented to demonstrate that our SMP approach for reinforcement learning can produce reliable control policy, and the gradient descent type optimization in the SMP solver requires less training episodes compared with the standard dynamic programming principle based methods.