Learning Stochastic Optimal Policies via Gradient Descent

TL;DR

This paper introduces a gradient descent-based method for learning stochastic optimal control policies by directly optimizing parametric controllers, extending classical SOC techniques and demonstrating effectiveness in portfolio optimization.

Contribution

It develops a variational calculus approach for stochastic differential equations and applies iterative gradient descent to optimize control policies under less restrictive assumptions.

Findings

Effective in continuous-time portfolio optimization with transaction costs

Extends classical SOC methods to broader functional forms

Demonstrates improved policy learning via gradient descent

Abstract

We systematically develop a learning-based treatment of stochastic optimal control (SOC), relying on direct optimization of parametric control policies. We propose a derivation of adjoint sensitivity results for stochastic differential equations through direct application of variational calculus. Then, given an objective function for a predetermined task specifying the desiderata for the controller, we optimize their parameters via iterative gradient descent methods. In doing so, we extend the range of applicability of classical SOC techniques, often requiring strict assumptions on the functional form of system and control. We verify the performance of the proposed approach on a continuous-time, finite horizon portfolio optimization with proportional transaction costs.