Model-Based Offline Planning with Trajectory Pruning

TL;DR

This paper introduces MOPP, a lightweight model-based offline planning framework that improves performance by aggressive trajectory rollout and pruning, addressing practical challenges in offline RL for real-world systems.

Contribution

MOPP is a novel offline planning framework that balances trajectory exploration and pruning, enhancing offline RL performance in real-world control tasks.

Findings

MOPP achieves competitive results with existing offline RL methods.

Trajectory pruning improves planning robustness.

Aggressive rollout guided by learned behavior policy enhances performance.

Abstract

The recent offline reinforcement learning (RL) studies have achieved much progress to make RL usable in real-world systems by learning policies from pre-collected datasets without environment interaction. Unfortunately, existing offline RL methods still face many practical challenges in real-world system control tasks, such as computational restriction during agent training and the requirement of extra control flexibility. The model-based planning framework provides an attractive alternative. However, most model-based planning algorithms are not designed for offline settings. Simply combining the ingredients of offline RL with existing methods either provides over-restrictive planning or leads to inferior performance. We propose a new light-weighted model-based offline planning framework, namely MOPP, which tackles the dilemma between the restrictions of offline learning and high-performance planning. MOPP encourages more aggressive trajectory rollout guided by the behavior policy learned from data, and prunes out problematic trajectories to avoid potential out-of-distribution samples. Experimental results show that MOPP provides competitive performance compared with existing model-based offline planning and RL approaches.