Supervised Learning for Stochastic Optimal Control

TL;DR

This paper introduces a method to generate supervised learning data for continuous-time nonlinear stochastic optimal control problems using the Feynman-Kac theorem, enabling the application of supervised learning techniques to control tasks.

Contribution

It presents a novel approach to automatically generate training data for stochastic optimal control by leveraging the Feynman-Kac theorem and stochastic process simulation.

Findings

Enables supervised learning for stochastic control problems.

Uses Feynman-Kac theorem to sample value functions.

Facilitates rapid data generation with hardware accelerators.

Abstract

Supervised machine learning is powerful. In recent years, it has enabled massive breakthroughs in computer vision and natural language processing. But leveraging these advances for optimal control has proved difficult. Data is a key limiting factor. Without access to the optimal policy, value function, or demonstrations, how can we fit a policy? In this paper, we show how to automatically generate supervised learning data for a class of continuous-time nonlinear stochastic optimal control problems. In particular, applying the Feynman-Kac theorem to a linear reparameterization of the Hamilton-Jacobi-Bellman PDE allows us to sample the value function by simulating a stochastic process. Hardware accelerators like GPUs could rapidly generate a large amount of this training data. With this data in hand, stochastic optimal control becomes supervised learning.