Reinforcement Learning Framework For Stochastic Optimal Control Problem Under Model Uncertainty

TL;DR

This paper introduces a continuous-time reinforcement learning framework that handles model uncertainty in stochastic optimal control by transforming the problem into an entropy-regularized control problem, enabling practical algorithms.

Contribution

It develops a novel entropy-regularized reinforcement learning approach for stochastic control under model uncertainty, using Sion's minimax theorem to simplify the problem.

Findings

Successfully applied to linear-quadratic problems with uncertain parameters

Established conditions for the validity of the minimax transformation

Demonstrated effectiveness on problems with Bernoulli and uniform distributions

Abstract

We develop a continuous-time entropy-regularized reinforcement learning framework under model uncertainty. By applying Sion's minimax theorem, we transform the intractable robust control problem into an equivalent standard entropy-regularized stochastic control problem, facilitating reinforcement learning algorithms. We establish sufficient conditions for the theorem's validity and demonstrate our approach on linear-quadratic problems with uncertain model parameters following Bernoulli and uniform distributions.