Convergence of the Value Function in Optimal Control Problems with Unknown Dynamics

TL;DR

This paper investigates the convergence of value functions in optimal control problems with uncertain, probabilistic dynamics, providing theoretical insights and numerical validation relevant to model-based Reinforcement Learning.

Contribution

It introduces a general framework for analyzing the convergence of value functions under probabilistic dynamics, bridging optimal control and Reinforcement Learning.

Findings

Theoretical proof of convergence under broad assumptions

Numerical experiments confirming the theoretical results

Framework applicable to probabilistic model-based RL algorithms

Abstract

We deal with the convergence of the value function of an approximate control problem with uncertain dynamics to the value function of a nonlinear optimal control problem. The assumptions on the dynamics and the costs are rather general and we assume to represent uncertainty in the dynamics by a probability distribution. The proposed framework aims to describe and motivate some model-based Reinforcement Learning algorithms where the model is probabilistic. We also show some numerical experiments which confirm the theoretical results.