Deep Model Predictive Optimization

TL;DR

Deep Model Predictive Optimization (DMPO) enhances control policies by learning the optimization process itself, leading to more robust and sample-efficient performance in complex robotics tasks like quadrotor flight.

Contribution

DMPO introduces a learned inner-loop optimization for MPC, improving robustness, efficiency, and adaptability over traditional MPC and model-free methods.

Findings

DMPO outperforms baseline MPC by up to 27% in performance.

DMPO achieves 19% better results than end-to-end MFRL policies.

DMPO requires 4.3 times less memory and fewer samples.

Abstract

A major challenge in robotics is to design robust policies which enable complex and agile behaviors in the real world. On one end of the spectrum, we have model-free reinforcement learning (MFRL), which is incredibly flexible and general but often results in brittle policies. In contrast, model predictive control (MPC) continually re-plans at each time step to remain robust to perturbations and model inaccuracies. However, despite its real-world successes, MPC often under-performs the optimal strategy. This is due to model quality, myopic behavior from short planning horizons, and approximations due to computational constraints. And even with a perfect model and enough compute, MPC can get stuck in bad local optima, depending heavily on the quality of the optimization algorithm. To this end, we propose Deep Model Predictive Optimization (DMPO), which learns the inner-loop of an MPC optimization algorithm directly via experience, specifically tailored to the needs of the control problem. We evaluate DMPO on a real quadrotor agile trajectory tracking task, on which it improves performance over a baseline MPC algorithm for a given computational budget. It can outperform the best MPC algorithm by up to 27% with fewer samples and an end-to-end policy trained with MFRL by 19%. Moreover, because DMPO requires fewer samples, it can also achieve these benefits with 4.3X less memory. When we subject the quadrotor to turbulent wind fields with an attached drag plate, DMPO can adapt zero-shot while still outperforming all baselines. Additional results can be found at https://tinyurl.com/mr2ywmnw.