Learning Continuous Control Policies by Stochastic Value Gradients

TL;DR

This paper introduces a unified framework for learning continuous control policies using stochastic value gradients, enabling flexible model-based and model-free methods that improve learning efficiency and robustness in control tasks.

Contribution

It develops a spectrum of policy gradient algorithms that incorporate stochasticity and learned models, reducing compounded errors and enabling simultaneous learning of models, value functions, and policies.

Findings

SVG(1) effectively learns models, value functions, and policies simultaneously.

Algorithms perform well on toy and physics-based control problems.

Framework unifies model-free and model-based approaches in continuous control.

Abstract

We present a unified framework for learning continuous control policies using backpropagation. It supports stochastic control by treating stochasticity in the Bellman equation as a deterministic function of exogenous noise. The product is a spectrum of general policy gradient algorithms that range from model-free methods with value functions to model-based methods without value functions. We use learned models but only require observations from the environment in- stead of observations from model-predicted trajectories, minimizing the impact of compounded model errors. We apply these algorithms first to a toy stochastic control problem and then to several physics-based control problems in simulation. One of these variants, SVG(1), shows the effectiveness of learning models, value functions, and policies simultaneously in continuous domains.