Data-Driven Stochastic Optimal Control in Reproducing Kernel Hilbert Spaces

TL;DR

This paper introduces a data-driven method for nonlinear stochastic optimal control using RKHS embeddings and operator regression, enabling scalable solutions for complex control tasks without explicit system models.

Contribution

It presents a novel RKHS-based operator learning framework that integrates with Hamilton-Jacobi-Bellman recursions for scalable nonlinear stochastic control.

Findings

Successfully applied to autonomous underwater vehicle depth regulation

Handles unknown nonlinear dynamics and costs using only control penalties

Scales linearly with state dimensionality in complex control tasks

Abstract

This paper proposes a fully data-driven approach for optimal control of nonlinear control-affine systems represented by a stochastic diffusion. The focus is on the scenario where both the nonlinear dynamics and stage cost functions are unknown, while only a control penalty function and constraints are provided. To this end, we embed state probability densities into a reproducing kernel Hilbert space (RKHS) to leverage recent advances in operator regression, thereby identifying Markov transition operators associated with controlled diffusion processes. This operator learning approach integrates naturally with convex operator-theoretic Hamilton-Jacobi-Bellman recursions that scale linearly with state dimensionality, effectively solving a wide range of nonlinear optimal control problems. Numerical results demonstrate its ability to address diverse nonlinear control tasks, including the depth regulation of an autonomous underwater vehicle.