Data-Driven Stochastic Optimal Control Using Kernel Gradients

TL;DR

This paper introduces a novel data-driven stochastic optimal control method leveraging kernel embeddings to efficiently approximate and optimize control policies using gradient descent in a reproducing kernel Hilbert space.

Contribution

It presents a gradient-based approach using kernel embeddings for stochastic control, enabling optimization over continuous control spaces with observed data.

Findings

Effective in linear regulation problems

Applicable to nonlinear target tracking

Demonstrates efficiency of kernel gradient methods

Abstract

We present an empirical, gradient-based method for solving data-driven stochastic optimal control problems using the theory of kernel embeddings of distributions. By embedding the integral operator of a stochastic kernel in a reproducing kernel Hilbert space, we can compute an empirical approximation of stochastic optimal control problems, which can then be solved efficiently using the properties of the RKHS. Existing approaches typically rely upon finite control spaces or optimize over policies with finite support to enable optimization. In contrast, our approach uses kernel-based gradients computed using observed data to approximate the cost surface of the optimal control problem, which can then be optimized using gradient descent. We apply our technique to the area of data-driven stochastic optimal control, and demonstrate our proposed approach on a linear regulation problem for comparison and on a nonlinear target tracking problem.