Combining Benefits from Trajectory Optimization and Deep Reinforcement Learning

TL;DR

This paper proposes a hybrid approach combining trajectory optimization and deep reinforcement learning to enhance sample efficiency and provide safety guarantees for robotic control, demonstrated on a car model.

Contribution

It introduces a method that integrates trajectory optimization with reinforcement learning to reduce sample complexity and estimate worst-case performance bounds.

Findings

Reduced RL sample complexity using optimal control knowledge

Provided upper bound estimates for time-to-arrival during training

Demonstrated applicability to mobile robotic systems

Abstract

Recent breakthroughs both in reinforcement learning and trajectory optimization have made significant advances towards real world robotic system deployment. Reinforcement learning (RL) can be applied to many problems without needing any modeling or intuition about the system, at the cost of high sample complexity and the inability to prove any metrics about the learned policies. Trajectory optimization (TO) on the other hand allows for stability and robustness analyses on generated motions and trajectories, but is only as good as the often over-simplified derived model, and may have prohibitively expensive computation times for real-time control. This paper seeks to combine the benefits from these two areas while mitigating their drawbacks by (1) decreasing RL sample complexity by using existing knowledge of the problem with optimal control, and (2) providing an upper bound estimate on the time-to-arrival of the combined learned-optimized policy, allowing online policy deployment at any point in the training process by using the TO as a worst-case scenario action. This method is evaluated for a car model, with applicability to any mobile robotic system. A video showing policy execution comparisons can be found at https://youtu.be/mv2xw83NyWU .