Theoretically Guaranteed Policy Improvement Distilled from Model-Based Planning

TL;DR

This paper introduces a theoretically grounded method for distilling model-based planning into RL policies, ensuring monotonic improvement and convergence, leading to enhanced sample efficiency and performance in continuous control tasks.

Contribution

It extends Soft Actor-Critic with a policy distillation approach from model-based planning, providing theoretical guarantees and a practical algorithm called MPDP.

Findings

MPDP achieves superior sample efficiency over existing methods.

MPDP demonstrates better asymptotic performance on MuJoCo benchmarks.

Theoretical analysis guarantees monotonic policy improvement and convergence.

Abstract

Model-based reinforcement learning (RL) has demonstrated remarkable successes on a range of continuous control tasks due to its high sample efficiency. To save the computation cost of conducting planning online, recent practices tend to distill optimized action sequences into an RL policy during the training phase. Although the distillation can incorporate both the foresight of planning and the exploration ability of RL policies, the theoretical understanding of these methods is yet unclear. In this paper, we extend the policy improvement step of Soft Actor-Critic (SAC) by developing an approach to distill from model-based planning to the policy. We then demonstrate that such an approach of policy improvement has a theoretical guarantee of monotonic improvement and convergence to the maximum value defined in SAC. We discuss effective design choices and implement our theory as a practical algorithm -- Model-based Planning Distilled to Policy (MPDP) -- that updates the policy jointly over multiple future time steps. Extensive experiments show that MPDP achieves better sample efficiency and asymptotic performance than both model-free and model-based planning algorithms on six continuous control benchmark tasks in MuJoCo.