Controlled Interacting Particle Algorithms for Simulation-based Reinforcement Learning

TL;DR

This paper introduces a novel simulation-based algorithm using interacting particle systems, specifically a dual ensemble Kalman filter, to approximate optimal control laws in continuous-time control problems, including nonlinear cases, without solving Riccati equations.

Contribution

It presents a new dual ensemble Kalman filter algorithm for control problems, enabling approximation of optimal controls without explicit Riccati equation solutions, applicable to nonlinear systems.

Findings

The dual EnKF algorithm effectively approximates optimal control laws.

Numerical experiments demonstrate the algorithm's accuracy and applicability.

Extension to nonlinear control problems broadens the method's scope.

Abstract

This paper is concerned with optimal control problems for control systems in continuous time, and interacting particle system methods designed to construct approximate control solutions. Particular attention is given to the linear quadratic (LQ) control problem. There is a growing interest in re-visiting this classical problem, in part due to the successes of reinforcement learning (RL). The main question of this body of research (and also of our paper) is to approximate the optimal control law {\em without} explicitly solving the Riccati equation. A novel simulation-based algorithm, namely a dual ensemble Kalman filter (EnKF), is introduced. The algorithm is used to obtain formulae for optimal control, expressed entirely in terms of the EnKF particles. An extension to the nonlinear case is also presented. The theoretical results and algorithms are illustrated with numerical experiments.