Direct Policy Optimization using Deterministic Sampling and Collocation

TL;DR

This paper introduces a novel approach combining direct trajectory optimization, deterministic sampling, and policy optimization to solve stochastic optimal-control problems, demonstrating effectiveness on linear and nonlinear robotic systems.

Contribution

The paper presents a new feedback motion-planning algorithm that integrates trajectory optimization, deterministic sampling, and policy learning, capable of handling complex control scenarios.

Findings

Recovers LQR policies for linear systems with Gaussian noise

Successfully applied to nonlinear, underactuated robotic systems

Handles control limits, obstacle avoidance, and unmodeled dynamics

Abstract

We present an approach for approximately solving discrete-time stochastic optimal-control problems by combining direct trajectory optimization, deterministic sampling, and policy optimization. Our feedback motion-planning algorithm uses a quasi-Newton method to simultaneously optimize a reference trajectory, a set of deterministically chosen sample trajectories, and a parameterized policy. We demonstrate that this approach exactly recovers LQR policies in the case of linear dynamics, quadratic objective, and Gaussian disturbances. We also demonstrate the algorithm on several nonlinear, underactuated robotic systems to highlight its performance and ability to handle control limits, safely avoid obstacles, and generate robust plans in the presence of unmodeled dynamics.