Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot

TL;DR

This paper presents a reinforcement learning approach that enables a robot to learn an effective control cost function from sparse objectives in under an hour, matching expert-tuned MPC performance.

Contribution

It introduces improvements to value learning methods that allow real-time deployment of RL for MPC on a real robot using only high-level sparse objectives.

Findings

The method learns cost functions from scratch without human tuning.

Achieves performance comparable to expert-tuned MPC.

Successfully deployed on a real unmanned ground vehicle.

Abstract

Model Predictive Control (MPC) is a powerful control technique that handles constraints, takes the system's dynamics into account, and optimizes for a given cost function. In practice, however, it often requires an expert to craft and tune this cost function and find trade-offs between different state penalties to satisfy simple high level objectives. In this paper, we use Reinforcement Learning and in particular value learning to approximate the value function given only high level objectives, which can be sparse and binary. Building upon previous works, we present improvements that allowed us to successfully deploy the method on a real world unmanned ground vehicle. Our experiments show that our method can learn the cost function from scratch and without human intervention, while reaching a performance level similar to that of an expert-tuned MPC. We perform a quantitative comparison of these methods with standard MPC approaches both in simulation and on the real robot.