Learning to Plan, Planning to Learn: Adaptive Hierarchical RL-MPC for Sample-Efficient Decision Making

TL;DR

This paper introduces an adaptive hierarchical RL-MPC framework that combines reinforcement learning and MPC to improve sample efficiency and robustness across various complex planning tasks.

Contribution

It presents a novel coupling of RL and MPC that adaptively guides sampling, enhancing performance and data efficiency in diverse decision-making domains.

Findings

Achieved up to 72% higher success rate than existing methods.

Accelerated convergence by a factor of 2.1 over non-adaptive sampling.

Demonstrated effectiveness across multiple complex planning tasks.

Abstract

We propose a new approach for solving planning problems with a hierarchical structure, fusing reinforcement learning and MPC planning. Our formulation tightly and elegantly couples the two planning paradigms. It leverages reinforcement learning actions to inform the MPPI sampler, and adaptively aggregates MPPI samples to inform the value estimation. The resulting adaptive process leverages further MPPI exploration where value estimates are uncertain, and improves training robustness and the overall resulting policies. This results in a robust planning approach that can handle complex planning problems and easily adapts to different applications, as demonstrated over several domains, including race driving, modified Acrobot, and Lunar Lander with added obstacles. Our results in these domains show better data efficiency and overall performance in terms of both rewards and task success, with up to a 72% increase in success rate compared to existing approaches, as well as accelerated convergence (x2.1) compared to non-adaptive sampling.