GPU-Accelerated Policy Optimization via Batch Automatic Differentiation of Gaussian Processes for Real-World Control

TL;DR

This paper introduces a GPU-accelerated policy optimization method using batch automatic differentiation of Gaussian processes, enabling scalable and efficient control policy training for real-world robotics applications.

Contribution

It presents a novel GPU-based approach that leverages batch sampling and automatic differentiation to significantly speed up Gaussian process policy optimization.

Findings

Achieved substantial speedup over exact methods.

Demonstrated scalability to larger policies and longer horizons.

Handled thousands of trajectories with minimal speed reduction.

Abstract

The ability of Gaussian processes (GPs) to predict the behavior of dynamical systems as a more sample-efficient alternative to parametric models seems promising for real-world robotics research. However, the computational complexity of GPs has made policy search a highly time and memory consuming process that has not been able to scale to larger problems. In this work, we develop a policy optimization method by leveraging fast predictive sampling methods to process batches of trajectories in every forward pass, and compute gradient updates over policy parameters by automatic differentiation of Monte Carlo evaluations, all on GPU. We demonstrate the effectiveness of our approach in training policies on a set of reference-tracking control experiments with a heavy-duty machine. Benchmark results show a significant speedup over exact methods and showcase the scalability of our method to larger policy networks, longer horizons, and up to thousands of trajectories with a sublinear drop in speed.