Value-Gradient Iteration with Quadratic Approximate Value Functions

TL;DR

This paper introduces a value-gradient iteration method for convex stochastic control problems that efficiently computes policies using quadratic approximate value functions, requiring minimal samples and tuning.

Contribution

It presents a novel value-gradient iteration approach that fits the gradient of the value function with regularization, enabling effective policy design with limited data.

Findings

Method finds good policies with low computational effort.

Requires few samples and minimal hyperparameter tuning.

Performs well even with approximate value functions.

Abstract

We propose a method for designing policies for convex stochastic control problems characterized by random linear dynamics and convex stage cost. We consider policies that employ quadratic approximate value functions as a substitute for the true value function. Evaluating the associated control policy involves solving a convex problem, typically a quadratic program, which can be carried out reliably in real-time. Such policies often perform well even when the approximate value function is not a particularly good approximation of the true value function. We propose value-gradient iteration, which fits the gradient of value function, with regularization that can include constraints reflecting known bounds on the true value function. Our value-gradient iteration method can yield a good approximate value function with few samples, and little hyperparameter tuning. We find that the method can find a good policy with computational effort comparable to that required to just evaluate a control policy via simulation.